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fS (57) Abstract: A method and system that uses a thin client solution in a mobile network is disclosed. The thin client (10) is not 
^ required to be equipped with an execution environment; rather, the client (10) is used as a display device for applications that run on 
^ remote servers (200). Applications such as E-mail client, browser and others execute on a remotely located server, but use the client 
^ (10) as a display and input device. The client (10) is equipped with a speech input device, which receives speech input and transmits 
^ it to the server for interpretation or recognition at-the server (200). Because bandwidth is limited, a method of combining requests 
^ that are transmitted and received between the client (10) and one or more server (200) is contemplated, which method results in a 
reduction of traffic between the client (10) and the server(s). The server (200), which runs applications that are used and accessed 
O by a user via the client (10), maintains application state on the server (200). Thus, when a user turns "ofiT* the client device (10), die 
^ server (200) may still maintain the state of the applications tiie user executed at the server (200). When die client (10) reestablishes 
>^ connection with the server (200), the user's prior state may be restored. 
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Technical FIELD 

The present Invention clainis priority from application 60^06,543 filed on May 23, 2000 titled 
REMOTE DISPLAYS IN MOBILE NETWORKS and application (no. pending) filed on March 18, 
2001 titled VIRTUAL PALMTOP PLATFORM. This invention is related generally to dient-^rver 
conriputing, arid particularly, to handheld computing devices used in mobile communication 
networi<8. 



Background 

_3Gls arieyo|ving wireless sta^ The types pfapplicatipris usedw 

include improved multimedia communications that combine voice, video, text and other methods. 
The next generation wireless communication services are expected to rely increasingly on 

IS data— including video and moving pictures— whereas the current wireless communication services 
are limited to voice-based applications such as telephone services. Other applications that are 
contemplated to use the new wireless networking standard include providing Intemet 
browsing— web browsing— in a large number of formats; and applications that enhance personal 
productivity— e.g., financial, calendar, groupware, and others— and location-based services. The 

20 current systems use microbrowsers and execution environments that run on the customer terminal 
. devices. The newly contemplated applications impose severe restrictions on the existing solutions. 
The amount of data that should be handled, the speed with which the data should be transmitted 
from and to the customer terminal device is expected to overwhelm the cun^ent architectures and 
strain the capabilities on both the customer devices such as the hand-held terminal devices as well 

25 as on the baci<-end devices such as the servers that support and deliver these applications. 

SUAAMARY 

This disclosure is directed toward a system and method to enable a client device such as a 
handheld computer or a cellular telephone device. Exaniples of such client devices include the 
30 commercially available Palm*^ Personal Digital Assistant (PDA), and similar devices marketed by 
companies such as Nokia Corporation of Finland, Handspring, Inc. of Mountain View, California, 
and others. The following discussion uses a novel device called a VirtualPalmTop. This device 
comprises hardware and/or sofiware configured to enable a general-purpose clierit device such as 
the aforiementioned exemplary devices to function in a manner as disclosed and described herein. 



A feature of these client devices is that they are small (i.e., they have a small screen or a 
fonn-fector). Because of their size, which is typically configured to fit In a person's shirt pocket, they 
cannot be equipped with an unlimited amount of memory. Further, their processing power is limited 
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because of limitations due to the amount of t>attery power available, and the amount of heat they 
can dissipate. A further feature of these client devices is that they are typically designed to work 
with a mobile communication network, and this requirement imposes additional restnctions such as 
the amount of bandwidth available, and how to handle in case of a service interruption or outage. 
Due to these and other limitations, applications that run well on conventional computers such as 
personal computers do not run as desired on these handheld.and/or mobile devices.. . 

Traditional systems such as the 3G PP Mobile Execution Environment (MExE) have been 
developed to enable execution of client applications on a client device for wireless and mobile 
applications. In contrast to MExE, one can execute applications on a remotely situated server, and 
use the client device as a mere output device rather than an execution environment By so 
configuring, the client device can be adapted to handle a number of applications that may be 
running on a variety of application servers. For example, a browser program may be mnning on a 
first server, and an electronic mail application may be running on a second server, but the client 
device will be used as a common output device for both the applications. As a consequence, the 
client device can be treated as a thin client with minimal need to handle complex applications 
locally. This configuration enables the client device to handle any type of application such as a 
Wireless Access Protocol (WAP) application, a Common Language Inftastorcture (CLI) application, 
or a Java Virtual Machine (JVM) application. Accordingly, with a minimal need for additional 
software on the client device, any cun-ently existing application may be accessed using this system. 

In some situations, however, there is a need to perfbmn certain localized actions on the client 
device as a result of a user action or inaction. Examples of these situations include the need to 
highlight a specified portion of the display area to indicate a user's selection of that portion of the 
display area. In such cases, there could be deposited on the client device a small piece of code to 
handle these localized actions without the need to transmit an indication of the user's action to the 
server and wait for an instruction as to how the user's action should be handled. As a result the 
amount of data transmitted from the client device to the sender (and back) is somewhat reduced. 

A second feature of the disclosed system is to further reduce the amount of data transmitted 
from the client device to the sender. This is accomplished by the use of a novel feature called a 
"compound request" In a nomnal client device, all user interactions generate "events." An event is 
typically a result of a user action or inaction. When events are generated at the client device, they 
are transmitted to the server, whereupon the server takes an appropriate previously defined action 
or may simply ignore the events. These events are sent to the server from the client device in a 
data packet But if a large number of data packets are generated at the client device, and all these 
packets are sent to the server, the limited bandwidth offered by the v\/ireless network may not be 
able to effectively handle them. Moreover, the transmission delay— also called "latency"— of these 
"evenf data packets decreases the responsiveness of the system and may lead to user frustration. 
Combining a number of events that occur in a predefined time Interval and transmitting them in a 
single data packet alleviates this problem. It should be noted, however, that these principles do not 
require thatonly one data packet be transmitted; rather, a fewer number of ^d 
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typically sent will reduce the latency problem. Similarly, a number of responses from the sender to 
the client device can also be combined and sent in a fewer number of data packets to save 
bandwidth. 

In order for the client device to provide a display interfiace to a number of applications that 
5 may run on a number of different application servers, It is necessary to maintain the state of each of 
... thesQ applications at some location, preferably on the respective servers. Advantageously, the state 

of the application programs may be stored so that in case of a disconnection between ttie client 

device and the server, a user may reestablish his state easily. 

10 Software Configuration on «ie Client device 

The presently disclosed system Includes a client device configured to support tiije above- 
mentioned features. It should be noted that a client device contemplated for use with ttiis invention 
does not need an execution environment, as is tfie case witii commercially available handheld 
computers such as Palm™. The presenUy contemplated dient device may use only a graphics 
15 protocol engine ratiier than a full-fledged execution environment Of course, it should be easily 
understood that in altemative embodiments, the graphics protocol engine could be added to a 
commercially available handheld computer, which may include an execution environment to function 
according to the mettiod disclosed herein. The description provided in this document presents the 
features of tiie client software, which features are collectively called VP client VP client features 
20 include, a graphics protocol subsystem, an event protocol subsystem, a speech protocol subsystem, 
and an ALM protocol subsystem. Also present is a cache management subsystem, which is 
configured to combine user actions and manage drawables and server requests. 

Application Perspective 

From an application environments viewpoint, a Virtual PalmTop Client device absb-acts its 
display as a series of drawables on which certain drawing actions are possible. The protocol 
between the client device and the server enables actions on these drawables and sending of user 
events back to tiie applications running on the server. 

Four different protocols come togetiierto define a system for rurining a broad range of 
graphical applications, multimedia applications, multimodal applications, etc. Fig 5 describes flie 4 
different protocols used by ttie invention that operate between clients and servers. The Graphics 
and Speech Interface Protocol specifies the mechanism of how the user display is updated and how 
multiple drawables on flie client are managed and the use of compound actions to update and 
manage fliem. 

The invention contains several different components. The Graphics and Speech protocol 
outiines a basic graphical interaction. There are several different capabilities in tills protocol. They 
are, (1) Capability Negotiation - Query tiie device for its capabilities and negotiate a list of 
capabilities to use in subsequent requests; (2) Mandatory Drawing Requests - a range of requests 
tiiatestaSiish the tT^^^^ (3) Optionah/Vldg^^^^ 
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optional requests that allow a server to use the Widget capabilities of a device if they exist; (4) . 
Drawable Management (5) On Action primitives; (6) Visual Objects and Management; and (7) 
Speech support and Notification. 

The Event System protocol delivers user events and input back to the application mnning on 
5 the server. A variety of input methods are possible, including Key and Mouse style devices, touch 
panels, eto. In addition, speech too can be an input method. The Event System protocol delivers the 
various inpute to the sender and these are used to send input actions up to the application. 

The ALM protocol along with DHCP Is used to bootstrap handsets back to their prior existing 
state notably by helping them reestablish connections to the user's running applications. They also 
10 offer a way of launching new applications. 

An implementation would involve a 2.5 Generation or 3"^ Generation Wireless WideArea or 
Wireless LAN network. Handsets would run the VP Client. Senders hosted at the Mobile Service 
Switching Center (in the case of the Wide Area Solution) would act as the VP Sen/er. The thin client 

IS would use these servers to run applications downloaded from the Internet or those provided 

specifically by the sen/ice provider An altemative emobodiment would involve a trusted Application 
. Service Provider which purchases networking that allows it to run the same applets and applications 
on behalf of the client. A third embodiment would involve an enterprise or other service provider 
owning and operating the VP sender. It is also possible for the server to be agented and a mobile 

20 RPC implementation allowing an agent to exist between the server and the client The agent would 
mediate requests and act as a buffering agent that bridges two networks, namely the wireline 
Intemet and the wireless mobile network. 

An altemative embodiment exists in Wireless LAN environments. In these environmente, the 
VP client and server interact over the Wireless LAN in a similar fashion. It is also possible tor a 

25 handset to use both environments. In this case as the user leaves the WLAN environment and goes 
over the wide area environment, the user can connect back to his previously mnning applications 
now using the wide area connection or vice versa. Advantageously, the applications may also be 
migrated. 

30 Software Configuration on the Server Computer 

As is the case with the client device, the server computer is programmed to execute a number 
of subsystems, each designed to interact with a similar subsystem on the client device. Accordingly, 
counterparts to the each of the several subsystems present in the client device, namely, the 
graphics subsystem, the event protocol subsystem, the speech protocol subsystem, fte ALM 
35 subsystem and the cache management subsystem, are present in the sender computer. In addition 
to these subsystems, the server computer also executes application programs for use by a user of 
the client device and optionally, an execution environment such as a Java Virtual Machine. 

An aspect of the present disclosure includes a method of compounding drawing requeste and 
— —other-transmissions from the sen/er. For example, certein display actions required to update'a 
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drawable can be aggregated into fewer actions and sent to the client, thereby conserving bandwidth. 
This can be impleniented in a number of ways. For example, rf one considers a display system to 
be a series of drawables, one can aggregate a series of requeste to draw an object on the client 
device and send the aggregated actions to the client in a fewer number of transmissions. 
5 Additionally, a feature of the presently disclosed system includes storing these compounded 

' - drawing requests in the server for later use. By thus storing the date transmitted to the client device 
at the sender, the server can maintein stete infonnation for the client device* In case the client 
device is turned off or othenA^se losf ite stete, the server can restore the client device's stete by 
retransmitting ^replaying") the client's stete infonmation to the client device. A number of 

10 transmissions to be sent to the client can be aggregated and sent as fewer transmissions, than 
ordinarily required. 

As will be explained in detail below, the sen/e^ 
Application List Manager (ALM) module. The Client deals with the ALM server. ALM server l^eeps 
track of the nuniber of applications currently run by the client, and a list of available applications. 

15 *The respective servers where these applications were running when the client Is tumed off store and 
maintein a log file for that client for that application. When client is tumed on, the ALM server 
notifies the client (or the client informs the ALM server) and the original client stete is restored. 

VP Client Soecification 

20 The VP Client specification requires a set of mandatory procedures and a set of optional 

procedures. The mandatory procedures are raw drawing primitives and primitives intended to 
esteblish the initial capability negotiation and connect the system. Additional primitives like Widget 
primitives, drawable manipulation, OnAcHon Primitives, Visual Objecte and support procedures for 
compound requeste are optional. 

25 The VP Client provides the necessary display abstraction needed by a server to first obtein 

information on the capabilities of the display notebly its torn factor, color capabilities, voice 
capabilities and input capabilities, support for compound requests, caching, widgete, eto. 
VPSYSTEMINFO, INITIALIZEGI^HICSYSTEM. INITIALIZEEVENTSYSTEM. GETVISUALINFO 
provide support for such capability negotiation. 

30 An illustrative sequence of packete that can esteblish a client/sender session can be described 

as follows: 



Sender ^VPSYSTEMINFO— » Client 

<r — ^VPSYSTEMINFO (REPLY) 

Server INITIALIZEGRAPHICSSYSTEM--^ Client 

^ INITIALIZEGRAPHICSSYSTEM (REPLY) 

Sender InitializeEventSystem ^ Client 
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Server — InitEventSystem (Evnf protocol) 
InitEventSystem (REPLY) 



Client 



Server <- — ^InitiaiizeEventSystem (REPLY) 



Client 



This initial exchange sets up the Client and serverto inte^ 

requests from the server requiring drawing actions on drawables or other display or output actions. 
When there Is input, the VP client picks up these events and delivers them to the sender using the 
DeliverNextEvent or DellverEvents requests of the Event Protocol. 

An External Data Representation (XDR) library on the client first decodes every sender 
request Thereafter, VP Client executes a conresponding drawing action. The server request can be 
either a simple or a compound request. If it is a simple request, then atter the drawing action is _ 
taken the client replies with VPOK or indicates the cause of the failure. If it is a compound request, 
the client executes the various subrequests contained in the compound request in a predetenmlned 
sequence and then returns the results In one large reply. If any of these subrequests faWs, the client 
may both stop executing additional requests and return with a failure indication and the list of 
committed requests. Alternatively, the client may ignore the felled subrequest arid continue 
processing the remaining submquests. The server can then replay the rest of the requests as It 
sees appropriate. 

A server will then use either a raw drawing protocol and/or a widget protocol. If the VP client 
and sen/er negotiate that the client supports compound requests but not widgets, then the VP server 
will look to issue a sequence of drawing actions corresponding to each widget drawing request. The 
following code Is an example of a graphical application in the Java programming language: 

Button b = new Button ("ABUTTON"); 
f.add (b); 

Scrollbar B = new Scrollbar (....); ... 
f.add (B); 
f.setVisible (); 

The above will cause ttte setVisible request to issue possibly one or two compound requests 
to the VP client This may lead to a single Compound Request (CR): 



DrawRect 
DrawLine 
Drawstring 



I ^ These represent CR for Button 



DrawRect | 
DrawRect - | 
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DrawPolygon 

DrawPolygon 

DrawRect 

DrawUne 

DrawLine 



->These represent CR for Scrollbar 



The ability to batch requests represents an enormous win for thin client solutions over wireless 
when compared to the use of single drawing requests both in minimizing overhead and reducing 
over -the- air latency. Individually and serially sent the 10 requests would take 1 second on a 

10 wireless network with 1 00 ms. latency as contention to regain control enonmously slows down the 
network with additional costs relating to interrupt latency eta adding marginal delay. Using the one 
compound request the thin client^solu^ _ 

A client can cache requests on drawables as it chooses but must do so on an all or none 
basis. This enables cache replay on a per drawable basis. The VP Server section provides an 

15 illustration for how efficiently drawable state can be reestablished by a VP server on a client. 

The VP Client tracks these drawables. A prefen-ed embodiment might store the drawable 
cache in Non-Volatile Memory but this is NOT a reqirement. At a time only one application is 
considered ACTIVE in a VP client The VP client could switch from application to application. As it 
does so, the drawables of suspended applications are deleted gradually to make space for new 

20 drawables for the active application. This form of lazy caching— i.e. . not discarding cache soon after 
it is no more needed or actively accessed — is also commonly used in microprocessor memory 
hierarchies, virtual memory systems and caches in file systems. The VP client which is close to the 
display where there is miriimal memory present is viewed as a more expensive cache for requests 
on active drawables. A discarded cache can be updated as and when required by the conesponding 

25 server side environment (like Java Virtual Machine) supporting the application. 

The VP client implementation can vary based on the capabilities of the device. On a device 
with a very good Windowing system and widget library a very advanced implementation is possible. 
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Brief Description OF THE Drawings 

These and other features of the principles of the disciosed systenr) are more readily 
understood fifom the following detailed description In conjunction with the accompanying drawings, 
In which, 

5 Fig. 1 shows a client device that can be adapted to practice the principles of the present 

-Invention. _ < . . . 

Fig. 2 depicts a server computer that can be adapted to implement the principles of the 
present Invention. 

Fig. 3 describes the 36 Environment depicting servers and clients Including several server 
10 famis hosted in the service environment at the mobile switching services center (MSG) by the 
Wireless Service Provider, or at an ASP or within an Enterprise. 

Fig. 4 describes the conceptual architecture for a client device configured according to the 
principles disclosed herein, which figure shows a plurality of applications running on remotely 
located servers using the client device for output (display), in addition to a depiction of activity from 
15 the client device transmitted to the server. 

Fig. 5 depicts drawable cache on an illustrative client device configured in according to the 
principles of the present invention, and three applications— one active and two suspended-^each 
with a set of drawables, wherein the solid colored drawables indicate full caches while the white 
colored drawables indicate cleared caches. 
20 Fig. 6 depicts a server and a client configured according to the principles of the present 

invention showing a request from the server to the client, wherein the server holds all of its 
drawables, and the client has cleared Its cache on some drawables. 

Fig. 7 pictorially describes aspects of certain protocols— VP Graphics protocol, Speech 
Protocol, Event Protocol. ALM Protocol, remote procedure call (rpc) and DHCP-nn addition to RTP, 
25 an optional protocol for speech and multimedia. 

Fig. 8 shows an Illustrative embodiment of server software implementation using the Java® 
programming language Abstract Windowing Toolkit (AWT). 

Fig. 9 outlines a sample conflguration for a multimodal server farm that uses speech, and display as 
nnethods of communication. 

30 Fig. 10 shows an illustrative interaction of input events in a multimodal (speech and display) 

application. 



Detailed Description 
35 Definitions: 

Pixmap: An off-screen buffer on which a graphics can be drawn. After drawing Into a pixmap, 
the graphics can be copied to a window and cause the graphics to appear on a screen If the window 
is visible. It should be noted that there is.no need to have a pixmap buffer; a graphics can be drawn 
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on a window directly. But using a pixmap helps a rapid update of the screen without repeating a 
series of primitive drawing operations. Pixmaps are typically used to store image data such as 
icons, logos that are loaded from adisic. These images can then be copied to a window. Lii<e 
windows, pixmaps are sender-side resources. 

Bitmap: A bitmap is similar to a pixmap, except that in a bitmap a single bit represents each 
pixel. In X-windows terminology, a bitmap is considered a pixmap with a depth of 1 . 

Drawable: A drawable is anything on which a graphics can be drawn. Dravi^bles include 
windows, pixmaps, and bitmaps. According to X-windows terminology, everything that can be 
displayed is a drawable 

Window: In X, a window is an actual visible drawable element, and Is nonriaiiy rectangular in 
shape, though this shape can be modified. Pixmap, on ttie qtherhand, is an off-screen chunic of ^ . 
memory to which pne can draw a graphic, but a pixmap is not visible. Windows can have child 
windows that are bounded by the parent window. Windows can be resized, moved, their maslcs 
changed, reparented to other windows, mapped (i.e., once a window is created, it must be mapped 
to mal<e it visible), unmapped (i.e., a window can be made invisible by unmapping it) and destroyed. 
Windows and pixmaps can be drawin using several primitive drawing functions provided in X. 
Among others, this includes drawing lines, rectangles, polygons, filled rectangles or polygons, arcs, 
filled arcs, copying one drawable to another drawable, etc. 

Events: An event is a happening on a window. This could be a user action such as a button 
press, button release, key press, mouse click, exposure of a window on a screen, resizing a 
window, moving a window or an object, or simply moving a cursor or a mouse pointer on a window. 
A window can be configured to select what events are captured or serviced by that window. The 
events a window can select are enumerated by turning on appropriate bits in an "event mask." A 
programmer can configure an unlimited number of events including some events based on a user 
inaction such as timeout after a window is exposed, the amount of time expired after a button is 
pressed and held before it is released, etc. When an event occurs, if it is masked, then it can be 
captured by the server (in the case of X-windows) and propagated to a predefined code segment 
called an "event handler." The event handler either ignores the event or performs an action 
responsive to the event. 

Server: The word "sen/er* is used in two different ways in this document. "Sender" as applied 
to a client-server hardware system implies a server computer that senhces a client device's needs. 
On the other hand, the word "sen/er* as applied to a user-interface ("windowing") environment 
denotes a program that mns on a coniputer with which a user interacts. In a client-sender computing 
system using an X-window type user interface environment the "X-sen^er is a program that ains on 
the client device and is configured to capture events that are generated at the client device and 
transmit them to an "application" running on the server computer. In this document, the word 
"server" should be understood based on the context in either of these two ways. 
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^ent An "agent" as used in this document is an intennediate computer tliat may controi, 
manage, mediate or coordinate ttte data transmission activities t)etween the client device and the 
server computer, in the disclosed system herein, the agent computer is an optional component 

Hardware architecture 

As illustrated in FIG. 1 a client device 10 configured according to the principles disclosed 
herein Includes a central processing unit 20 ("CPU"), which could be a general-purpose processor 
such as an Intel® StrongARM® processor or a special-purpose processor. The CPU 20 Is 
connected through a bus 30 to, among other things, volatile memory 40 (also called RAM or random 
access memory), non-volatile memory 50 (such as disk drives, CD-ROMs or data tapes), a network 
communications adapter 60 (such as an Ethernet card), an Input means 70. such as a keyboard 
and/or a pointing or polnt-and-dick device (such as a moirae, light pen. touch screen, touch pad, joy 
stick, jog dial), an output device, such as a video display screen and/or an audio speaker, and a 
removable media drive 80, such as a floppy disk drive. CD-ROiVi drive, PCi^lA port, CD-WORM 
drive or data tape drive. 

The client device 10 operates client software 90 for use with the present invention. The client 
software is shown graphically in FIG. 1 as being stored in non-volatile memory 50. However, it 
should be understood that it can also be stored in transportable media read by removable media • 
drive 80. All, or portions of the client software 90 also can be loaded into volatile memory 40 (RAM), 
for example during operafion. Exemplary transportable media implementing the client software 
(which may be in any forni, such as source code, compiled or binary versions) include floppy disks, 
magnetic tape and optical disks, and others. In one embodiment, a client device is a portable 
computer such as a hand-held device and the electronic communications network is a wireless 
networi< connected to the Intemet or an online senflce. Further, "Client device" or "Client device" 1 0 
shouW be understood in this descriptton to include any portable or hand-^eid device used for access 
an electronic communications network, such as a data processing system. 

The client is optionally equipped with a focatfon-detemninatlon device (not shown In Rg. 1) 
such as a Global Positioning System (GPS) receiver whereby the physteai location of the dlent 
device, i.e., its coordinates or other infomiatlon that alfow one to locate where the client device is 
located. With the help of such location detemiinatfon device, the location of the client device can be 
detemiined and the location infomiation can be transmitted to a remote server. Using GPS 
infonnation, location-based sendees can be provided to the client Other methods of deteiminlng 
location include known methods such as triangulation such as those used in locating the position in 
cellular telephony applications may also be used. An advantage of determining location is that a 
user may see a different set of applications based on whether he/she is In a shopping mall, an 
airport, or a different location. 

. The client device is also equipped to receive speech input from a user.. A microphone can be 
incorporated to accomplish this (not shown in Fig. 1 ) or other speech input device such as a 
telephone into the client device; Speech can take two fomTs:-(1) either it can-be-speech-as a paft of 
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the payload such as in the case of a telephone call, i.e., when the client device is used to make 
ordinary telephone calls, or (2) when speech is provided as a way to instmct an application, for 
example, in the case of an E-mali application, a spoken input may be used to command the E-mail 
application to send a reply to an E-mail. 
5 Some embodiments described herein are related to methods to achieve better resulte in 

— wireless hand-held devices. In addition, such^ client devices may effectively employ the present 

invention when utilizing a variety of operating systems or programming languages, such as the 
"Wlndows-CE^ "Windows ME". Talm OS", "Linux" or other operating systems such as reakime 
operating systems. 

10 FIG. 2 is a block diagram showing a server computer 200. The sender computer 200 includes 

a general-purpose microprocessor 210 such as a IBM RS/6O60™ processor, a storage device such 
_ as a hard disk 220, memory device 230 such as senikx>riduct^ 
device 240 that enables the server computer 200 to connect to a network such as a wireless 
communication network 250. The server computer 200 is further connected to other data 

15 communication networks 260 such as the Internet— also known as the World Wide Web— and other 
public and private networks. The client device 10 is illustratively coupled to the sender computer 200 
via a mobile communication network such as a 3G networl< 250. See Fig. 3. In operation, the 
processor 210 operates software such as server software 270 adapted for establishing a 
communications link with the client device 1 0. Operation of the server software 270 is discussed in 

20 more detail below. Exemplary transportable media implementing the server software 270 may be in 
any form, such as source code, compiled or binary versions. The server computer 200 and the 
client device 10 may be programmed to execute instructions that instruct the respective processors 
to act in accordance with the principles disclosed herein. 

As steted above, the client device is coupled to the server computer using a wireless 

25 communication link. This link is explained below in the section 1.1.3 entitled Overview of the 
Speech Interactions and Protocol. 

The following describes one or more Illustrative embbdimente of the principleis discovered. 
These illustrative embodinriente should not be used to limit the spirit and scope of the novel 
principles disclosed. 

30 

The VP Servftr 
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maintain complete caches of all drawables and look to maximally use compound requests by 
aggregating various drawing actions. An implementation must look to perfonn several drawing 
actions at once. For instance, all drawing actions to a given panel that contains multiple widgets like 
buttons, text areas, etc. can all be aggregated into one compound request 

Fig. 9 outlines a simple configuration for a server fann. The Application on the VP server 
using an interface like Java^eec/i, handles speech input The Speech Recognition sender 
receives user input from the Event System proxy (or altematively from an RTSP connection with the 
handset) and following recognition returns the recognized speech back to the VP Server where the 
application would take appropriate action. An illustration - a user gesture (through touch), pointing 
to Australia, with a voice command, "Airport Locations", will result In the event proxy sending the 
touch gesture directly to the VP Sender and there on to the application, with the voice infomiation 
sent to the Speech Recognition server which following recognition is sent to the Application, which 

then outputs. "Sydney. Melbourne. Canbenra, Perth " on the map. This fomi of multimodai 

application is possible with VP as the application runs in the network. 

The VP servers in a server fami configuration on an MSG would implement the ALM protocol 
and run various applications. The ALM protocol implementation will wait for requests from a VP 
client When the VP client contacts ALM, it will look to either launch the ALM Ul application on a VP 
server that ALM specifies or directly connect to one of the existing applications that the user left 
ojnning on this or other server/ 

Refenring now to Fig. 5, the VP server running this application then examines the drawable 
cache on the client and replays the cache on the appropriate drawables, which are either stale or 
discarded. The server then issues a command LISTDRAWABLECACHE, which causes the VP 
client to return the list of drawables available. The VP Server then updates the discarded drawables 
using single large compound requests as shown in the example below. It then does a REPAINT 
DRAWABLE or SHOWDRAWABLE to bring the Client VP display state back into sync. The 
application then proceeds as the active application for this VP. 

For example, consider the following application, 

Abutton = new Button 0; 

AtextArea = new TextArea ("Enter text here... ."); 

Ascrollbar = new Scrollbar (); 

setVislbleO; . 

The AWT implementation would then take action to display the frame on the display. The 
technology at the time of setVisible must perform all the drawing operations associated with 
Abutton. AtextArea and Ascrollbar all at once using one single compound request This large 
compound request would be able to reduce the traffic to fewer— in the example shown, a single 
exchange^-thereby conseifvlng bandwidth and p^^^^ 
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Cache replay also can be in the form of compound requests if the VP dlent supports It This 
will allow a large set of requests to be replayed all at once. The mechanism for a REPLAY could 
make the best use of compound requests using the stacking paradigm altowed for drawables. 
Thus, an original sequence of requests - 

GREATEDRAWABLE->DRW1 

CREATEDRAWABLE -> DRW2 



I DRAWRECTDRW2 

C FILLRECTDRW2 

I COPYAREA PRW2 DRW1 



I FILLRECT DRW1 

C DRAWPOLYGON DRW1 

I DRAWPOLYGON DRW1 



C DRAWIMAGE DRW1 
I FILLRECT DRW1 



VVould be replayed as follows as 1 compound request 

I CREATEDRAWABLE 

I PUSHDRAWABLE DRW1 

i CREATEDRAWABLE DRW2 

I PUSHDRAWABLE DRW2 

I DRAWRECT (CURDRAWABLE) 

C FILLRECT (CURDRAWABLE) 

I COPYDRAWABLE (CURDRAWABLE) (CURDRAWABLE -1) 

i POPDRAWABLE 

I FILLRECT DRW1 

I DRAWPOLYGON DRW1 

1 DRAWPOLYGON DRW1 

I DRAWIIVIAGE DRWI 

I FILLRECT DRWI 
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This is a simple implementation illustration, which shows how efficiently drawables can be 
replayed using the mechanisms available In Virtual PalmTop. 

5 Event System Impleiyi^ptation 

-Refening to Fig. 10, the event system Implementation on the server can either use an agent 

or it can be implemented directly. The application environment implementation (for instance the 
Java AWT implementation in the case of Java) receives its events from the VP client either directly 
or through the event system proxy. A typical implementation on a server is the server side of the 
10 event system protocol (evntx) and it picks up user events and delivers in into the application 
environment which finally fonvards It into the Application. 

, The following illustration on Palm OS indicates an implementation^ 0^ 

static void 

EventLoop (void) 
15 { 

EventType event; 
Word error; 

do{ 

20 EvtResetAutoOffTlmer (); 

svc^run (); 

EvtGetEvent(&event, 0); 
25 switch (event.eType) { 

case appStopEvent: 
case nilEvent 
breaic; 

case IceyDownEvent 
30 // send hard-button presses directly to server . 

swftch (eventdata.keyDown.chr) { 
case pageUpChr case pageDownChn case vchrHarcH : 
case vchrHard2: case vchrHardS: case vchrHard4: 
pickup_event(&event); 
35 continue; 

} 

// fall through! sorry for the mess. 

delault: 



-14- 



wo 01/91482 



PCTAJSOl/17274 



// Give the system a chance to handle the event 
if (ISyaHandleEvent (&event)) { 

if(lgEventSyslnitialized){ 

5 

} else { 

If (!pickup_event (Aevent)) { 

} 

10 } 
} 

} 

^ } while (eventeType != appStopEveht); ^ 

}//end EventLoop () 

boolj 

plckup^event (EventPtr eventp) . 
{ 

NextEvent nevt; 
20 EVstat*retval; 
int i = 0; 

switch (eventp->eType) { 
case penDownEvent 
25 nevtEventDID = 1; 

nevtEventSeqNum = seqnum++; 
nevtevLetype = TOUCHEVENT; /* BUTTONPRESS */ 
nevtevtEvent_u.tps,x = eventp->screenX; 
nevtevLEvent_u.tps.y = eventp->screenY; 
30 while (f (retval = delivemexteventj (&nevt. gEvtcInt)) && (i++ < 5)) {} 
return tnie; 
case penUpEvent 
nevtEventDID= 1; 
nevtEventSeqNum = seqnum++; 
35 nevtevtetype = TOUCHEVENT; /* Note this was a BUTTONRELEASE */ 
nevtevt.Event_u.tps.x = eventp->screenX; 
nevtevt Event_u.tps.y = eventp->screenY; 
while (! (retval = delivemexteventj (&nevt, gEvtcInt)) && (i++ < 5)) {} 
return taie; _ . 
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case keyDownEvent 
nevtEventDID= 1; 
nevtEventSeqNum = seqnum++; 
nevtevtetype = KEYPRESS; 
5 nevtevLEvent_u.kps.x = eventp->screenX; 

nevtevtEventju.kps.y = eventp-^scrre - ' 

nevtevtEvenLu.kps.key = getkey (eventp->data.keyDown); 

while (! (retval = dellvemextevenM (&nevt. gEvtdnt)) && (i++ < 5)) Q 
10 return tme; 
default 
retumfalse; 

} 

} 

15 ' 

EVstat* 

delivemextevent_1 (aigp, cint) 
NextEvent*argp; 
CLIENT *clnt; 

20 { 

Static EVstat res; 

bzero ( (char *)&res, sizeof (res)); 

If (clnt^call (clnt, DeliverNextEvent, (xdrprocJ)xdr_NextEvent (char *)argp, 
(xdrprocJ)xdr_EVstat, (char *) &res, TIMEOUT) != RPC^SUCCESS) { 
return (NULL); 

} 

return (&res); 



EventLoop above waite for user events. When user events are received a remote procedure 
call to do a DELIVERNEXTEVENT or a DELIVEREVENTS is made this causes the event to be 
delivered to the server either direcHy or through a proxy. The server reacts to the input event and 
updates the screen correspondingly. 

Speech Proc essing CaDabllltv Imptementatiori 

There are many possibilities when speech is used in multimodal applications. They include 
the following input options - 

(A) Speech is used in a separate voice channel (through a phone call) 
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(B) Speech is used through the data channel but using a separate stream 

(C) Speech is used as part of the event system protocol as described in the VP client specification. 

(D) The event system can be proxied and speech can be delivered through this proxy. 

Speech input is then recognized and fonvarded into the application to indicate user requests. 
5 Speech output on the server can eitiier be raw or encoded speech in some of the indicated 

formats. ____ 

Fig 8. Describes how speech data when it is sent as part of the event system protocol is 
received by the proxy and fonvarded to the recognition engine, recognized and then fonn/arded to 
the end VP sender where the application resides which then takes appropriate action and updates 
10 the User Interfece. 

The various events that talce place from A through F in that order are as follows. The user 
simultaneously cliclcs and speaks (in the Illustration, he points at Australia and says, "Airport 
Locations"). The two user events are sent in two separate DeiiverNextEvents to the Event System 
Proxy which then fonvards the Event System request to the VP Server and the speech request to 
15 the Speech Recognition Engine. The Recognition Engine could request the end user application for 
strings to match against (this happens from inside the virtual environment say from within a 
java.speech Implementation). The application then sends a list of strings it is waiting on (or indicates 
that it receives all data). The Recognition engine matches the input to the provided strings or 
recognizes it othenA^ise and then fonvards the infonnation to the VP sen/er. (ALM too could be 
20 waiting for speech input to launch new applications, etc.). 

Multimedia Services 

In the case of multimedia a viewer running on a VP Server can selectively choose streams to 
send to the end user. For instance, in the case of MPEG-4 Audio Visual Objects (AVOs) are 
25 streamed separately. The background of a tennis match and the players and the ball each can be 
separate AVOs. With VP as the application runs on the sender and wireless bandwidth is expensive 
compared to wireline. It is possible for the application (the viewer) to selectively stream only the 
players and the ball and not the background. The viewer can cache all the streams allowing the user 
to choose to replay them. This form of capability Is possible easily with VP. (and is a requirement for 
.30 the Virtual Home Environment). 

1.1 Overview of the features Included in the client device 

Refening to Fig. 4, the client device supports multiple Input/Output features to enable remote 
servers to "mn" a variety of applications on the client device. The software configured to operate the 
35 Client device is called VP client, which includes a graphical display with support for at least raw 
drav/ing and simple graphical actions on the frame buffer. The VP client must also support at least 
one of mouse, keyboard, touch panel or similar input and can have multiple such input methods 
including speech. The ability to support speech input and output in any of the defined fomris is purely 
optionalrVP client can be compliant without any speech support although applira^^^ - 
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speedi as the sole input method or that mandate it might not "run" on devices that do not provide 
support for it. 

The Ciient device is intended to ain over simpie mot)iie transports where iatency is noticeably 
larger than equivalent Wire line counterparts. Optional features in the VP Protocol enable the 
efficient use of both bandwidth and the amelioration of problems relating to latency in Mdblte 
Networks by allowing for both caching of requests, their replay from these caches and the judicious 
.use of larger compound requeste. Wireless latencies may be in the range of 50 ms. to 100-ms for 
air interface for WCDI\A^. GPRS, UMTS and other networks. 

Client devices can also be used in WLAN environments. The availability of more bandvirtdth 
and the lower latency of these environmente further enhance the suitability, capabilities and 
appropriateness of using VP in educational environments, open public places, enterprises, small 
offices and tiulldings and homes. 

The Ciient device also can work on a range of devices with more complex and less complex 
capabilities. The onus of supporting a variety of these devices reste on the ability of sen/eis. 
Servers can however indicate that they are unable to support certain features and the VP Client 
must then fell back to find a least common denominator of its own and the sender's capabilities. 

The VP Client can support muitiple features including Speech and multimedia. In the case of 
speech, it is possible for speech to be delivered to servers either on a separate channel, over the 
data channel with no detection of silences (on a continuous basis) or as and when the user speal®. 
The fonnat of speech too is detemiined at the time of capability negotiations, in a multimodal 
application; the speech can be played back either using a TTS converter on the dient device or by 
having tiie server stream out the speech over the negotiated channel. 

VP Cliente are also intended to "run" server hosted MMS applications, MPEG^J based 
applications, and bring about a rich environment of multimedia«nabled applications for mobile 
users. White this is optional on VP Cliente. flie ability to support multimedia is part of the VP Client. 
Further it should be possible for users to selecBvety turn off some of «ie channels of a multimedia 
message replay as and vi/hen they want (A user might want to tum off the video but not the audio of 
a message to save bandwidth while on the road but not white he Is at his office). 
VP Cliente can run mutlipte appiicattons concunentfy. In one embodiment there could be only one 
application active at a time (this is implied but it is not mandated). VP Cliente can obtain information 
about any or all applications at any time. 

At the time of bootetrap the VP Client obtains session, seairity and state InfonnaHon about 
the User's current list of Applications and verifies login and authenticates the user. Foltowing 
authentication, the VP Client is bootetrapped back to Ite prior existing state. The Application List 
protocol (along with DHCP) is intended to provide support for this process and is implemented by 
the Application List IWIanager. It could contact an Application Ust Manager Ul, which Is a VP 
Appliration hosted by the bootetrap sen/er that drives tfie User Interfiace of the VP Client with an 
application from where the user can launch additional applications and brovi«e his cunent Virhiai 
Environment-Additionally, as part of me Bootetrap proce^^ 
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RTP/RTSP session to a proxy server. As the VP Client changes Application Servers, information 
about this proxy server and the VP Clienf s session with it are forrvarded to each new Application 
Server. 

5 1.1.1 Overview of Graphics Interactions and Protocol 

. . The GraphjcsJnteractionProtocoMs Intended to be a drawing protocol that 
graphical capabilities of a gamut of device classes. It ranges from allowing very simple devices to 
implement a very simple graphics protocol that uses more bandwidth and have higher latency to 
more complicated devices that have inherent Widget support and .can handle compound requests 

10 and cache them thus using lesser bandwidth and working well over high latency environments. The 
Graphics Interaction Protocol allows the client and server to negotiate the capabilities for the 
session at the outeet allowing them to specify the right set of features that would be used in the _ 
session as well as when the client is reconnected to the server. The goal of the VP Client is to 
provide the richest possible user support given the capability negotiation. The VP Client must 

15 attempt to conserve bandwidth and decrease end user latency and implementations are encouraged 
to support this. 

The Graphics Interactions protocol works by providing drawing and rendering actions on 
drawables. These are flat rectangular surfaces that vary in size from the size of the screen of the 
client device (full drawables) (or possibly even larger surfaces if the VP client supported this 

.20 capability) to subdrawables that are smaller. Drawables can nest (only one level of nesting is 

supported in this version of the protocol) and there can be drawables that can be contained inside a 
full drawable and allow independent actions on them. There is the notion of a root drawable that 
represente the physical screen, as well as off-screen drawables that represent memory for 
drawables, that can be shown as needed. Drawables need not persist across disconnections. 

25 However, there could be clients that could support persistence. (The onus reste on servers to 
support both persistent and non-persistent drawable systems). The Client device at the outset 
indicates the number of drawables it can reasonably support and the server will try to restrict ite 
usage to within that number. A variety of operations are possible on these surfaces and they forrri 
the basis of this protocol. 

30 

1.1.1.1 Compound Operations on Drawables 

The goal in providing support for Compound Operations is to decrease the number of over the 
wire operations and interchanges. This decreases the associated end user latency while reducing 
contention over a potentially shared air interface. A compound operation may take place on one or 
35 more drawables. But each element of a compound request will correspond at most to one or two 
drawables. They would however have to involve one target drawable (either a root drawable or an 
off screen drawable). Operations to copy drawables, copy an area of a drawable, etc. also are 
possible. The aim in such operations is to reduce the amount of over tiie wire date as well as to 
allow reduced lateney^Further cliente can cache operations on a drawable without'actiMlly 
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rendering them In the case of off screen drawables (but not the display itself). Client behavior does 
NOT require the creation of dnawables at the time of the call. Instead the system can choose to 
postpone actions based on available memory capacity, etc. till the time when it is deemed 
necessary to render it to the off screen or when the drawable Is shown (rendered to the display). 
5 To support compound operations, there are two drawables that are defined. There Is the 

- cynrent drawable of ihe compound request, which is passed Ijetween the individual requests In the " 
compound request This could be the result of a CREATEDRAWABLE request or passed as the 
drawable In the first request. There is the saved drawable which Is pushed into a save area. The 
definition of the cunent is NULL and the saved drawable is -1 ; (fffffffF) 
0 This Virtual Screen system thus allows the ability of a limited memory system to support . 

multiple actions on a drawable. Further, as Indicated in the protocol, it is possible to have the VP 
client indicate that it no longer has state associated with a specific (off screen) drawable (which 
must be uniquel). The server could then replay with the actions on the drawable and then repeat the 
request 

5 Supporting Compound operations is recommended but not mandatory. A limited system could 

choose to not Implement Compound Operations and might work just as well In certain 
environmente. 



11.1.2 Client Capabilities and Negotiations 
20 Client Capabilities are negotiated at the beginning of each application session. Senders may 

choose to reuse Client Capability infomiation to save bandwidth. However, it must indicate the start 
of a new application as Cliente use this information to purge and reuse cached data using their own 
smart algorithms. 

Client Capability Negotiation relates to a large number of display and input features as well as 
25 system features. These include whether the system supports color displays, input features like 
speech, font capabilities, image formats, multimedia support, etc. 

Client Capability Negotiation allows servers to negotiate the full capability up front This allows 
Sen/ers to plan their actions accordingly. For instance, a sen/er might use this client capability to 
dedde whether It will support compound operations, drawing interactions, etc. 

30 

1.1.1.3 The State of VP Cliente and their relationship to sessions 

A VP Client can be In one of several states with respect to sessions an ACTIVE state of the 
session, which is when an application is actively running; and a STANDALONE state when there is 
no active session at a given instant State of a VP Client is defined within the context of a session 
35 with a server. A session is said to be ACTIVE If an application Is active and controls the display at a 
particular instant When* the VP Client transitions from flie ACTIVE state. It could eitiier go to a 
SUSPENDED state, a DISCONNECTED state or a TERMINATED state for that session. The 
SUSPENDED state is when the VP Client transitions from one to anoflier application. This second 
application could be ninning onieilther tile ^me serverbr on another server. Whenln the "~ ~ ' 
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SUSPENDED State, the VP Client can choose tp disconnect either by sending a message or by 
brealdng the connection (if there is one). A server does not know If the client has transitioned to the 
DISCONNECTED state or not unless it has explicitly said so. Thus servers wait for prespecified 
periods of time (recommended 4 minutes) before assuming that they are disconnected. A sender 
5 caches ALL information needed to restore state. This includes actions perfonmed on offscreen 
_ _ drav^bles, the root drawable as well as other infomiation. The TERIWINATED state Is caused by an 
exiting of the Application or by the user initiated tennination (or killing) of the application (the client 
deletes all caches). 

A client which supports Compound requests must cache requests on the cunent active 
10 drawable. it must also look to cache requests on as many drawables in the ACTIVE session if 
possible, but if it cannot, it must respond with a REPLAY enror during the time of a 
SHOWDRAWABLE on a drawable. This allows the system to reconstruct its state on the drawable, 
which is then shown. 

Clients have no ability to command reconstruction of state. They are forced to rely on servers 
15 to help them. At the time of reconnection (implying a move from either the SUSPENDED state or the 
DISCONNECTED state) back to the ACTIVE state a server might request the list of cached 
drawables from a client and choose to update them. This might help speed the user response time 
subsequent to reconnection. 

20 1 . 1 . 1 .4 ColorSystems 

The various color systems that VP Clients could support include the Gray Color, RGB, YUV or 
RGBA systems and other systems that represent extensions. RGB systems can be either RGB565, 
RGB888. VPCIients could support any of the defined colorsystems as well as additional color 
systems that are specific to the server and the client It is hoped that these ColorSystems will be 
25 standardized (through addenda). 

Further there is support for other colorsystems and one can use GETVISUALINFO for 
obtaining visual infomrtation. However VP Clients must support atleast one of the defined 
colorsystems in addition to any extensible colorsystems. 

30 1.1.1.5 Drawing Interactions 

A varieV of drawing interactions are specified in the protocol. They include the drawing of 

common graphical shapes including rectangles, polygons, ellipses and arcs, lines and solid objects. 

Drawing interactions do not reflect the needs of 3D. This will be tackled in later versions of the 

protocol. The client specification requires that the handset should support all mandatory procedures 
35 (18 to 31) shoyvn in 5.18 to 5.31 to be in compliance. The client should support this minimal set of 

procedures. 

Support for Drawables is mandatory. At least one offscreen drawable and the ability to create 
sub-drawables within the main screen must be provided. The recommended number is at least 4 
"Offscreen drawables for very limited membfy devices and 8 of fffdre drari^ for devices vwtfi 8 " 
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MB or more available memory supporting compound operations with cacliing and wlien there is 
support for multiple simultaneous applications. The intention Is to allow these client devices to 
maximally use their networked capabilities to bring greater user access and productivity in a 
dynamic mobile environment 
5 Additionally the interactions do require that when support for "OnAction" operations (including 

mouse over operations) exists, the Implementations do make a best effort to "render the changes as 
reflected, (there is no drag and drop support in this vereion of the protocol). Support for OnAdion 
operations is optional. 

10 1.1.1.6 Font Handling Interactions 

The support for font handling is similar to drawing interactions. At «ie time of the capability 

. r®SPtiations, all supported fonte must be indkated^ .. 

the appropriate font for rendering on the screen. Support for scaleable fonts is an important (ttiough 
optional) requirement for VP devices. If this is absent it is difficult to support Classmark 11 and 
15 Classmark III devices and possibty other future classmarks. 

The font handling system could also be addifionalty compliant with "VP Font Metrics". This will 
allow servers to avoid querying the client device about the lengths of text sequences ttirough 
"GetTextWIdth" calls. 

Additionally Font Handling Interactions could support OnAction operations (to Underline, 
20 Embolden or Itelicize). This is optional. 

1.1.1.7 Image Interactions 

At the time of flie capability negotiation, ttie image interactions are determined. The VP Client 
must spedty the fonnats it can encode indicating as many fonnats as it can. This will allow tiie 
25 seiver to detennlne tiie appropriate fonnat for transmission. It may be ttiat Oie sender might use onty 
one of tiie formate specified for over ttie air tiansmission. The type of image used is specified per 
image tiansmission. Support for ttie improvement of visual clarity might be present in servere and 
diente (if tiiey botii support fliis feature). In fliis case, at ttie user's behest, tiie system can choose to 
rebansmit ttie picture possibty witti lower compression or in an Improved format 

image Interactions are Intended to simplrty ttie easy handling of Images and rendering of 
compressed and uncompressed pictures over ttie air. The image system is lossy In ttiat ttie picture 
Isrecompressed (ttiis is avoided if tiie final fomiat supported and ttie fom^at of ttie original pkSure is 
the same) in ttie negotiated fomnat 

35 1.1.1.8 Widget Interactions 

The aim in providing support for Widget Interactions is to allow stete of ttw art operating 
systems ttiat have compact Widgets to make very effective use of tiieir capabilities. The system at 
ttie time of capability negotiation indicates support for Widget Interaction. The Widgets are tiien 
^^'^"^'ered on ttie display. TijayifBupdatid and M^^^^ 



30 
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Widget, including selections and are sent back to the server. Selections are sent only after the user 
makes one and not while he scrolls through the list Typically servers could also maintain a mirror of 
the client's Widget and reflect actions on it 

Scrolling actions are not fypically sent over the wire until the final state is reached at which 
5 point a "Scrolled ToTexT request is sent 



1.1.1.9 Advanced Usage of Compound Operations 

The usage of Compound Operations tends to make things difficult for Servers. For this reason 
it is important to make client behavior specific for each sequence of compound operations. 

10 Compound operations tend to produce many failure cases as any one of a sequence can fail. 

Sequence numbers give a server a chance to recover finom failures due to over the air corruption or 
errors. However, Client behavior must strive to respond predictai?|y.^ This allows Servers to make 
best case efforts and assumptions. 

VP Client behavior requires that clients stop perfomning subsequent operations in a sequence 

15 of corrqiound operations if one operation fails. This behavior allows servers to recover by looking at 
the enor on the ^iled request and replaying subsequent requests in the set 

Further, VP Clients must cache Compound Requestsrmade to on screen drawables. This is 
required due to OnAction style operations. 

In the case of OffScreens of the current active application, the client must request a 

20 REPLAY of all requests to a drawable at the time of a SHOWDRAWABLE and reach the state in 
which the drawable was at the time of the request to SHOWDRAWABLE. 
It is typical that this implies that Clients cache either all or no operations oh any drawable. The 
intention must be to try and preserve over the air bandwidth and user latency. 

A further use of advanced compound operations involves the definition of visual objects and 

25 movement actions on them. Support for Visual Objects is a strictly optional feature for a VP Client 
A Visual Object is a persistent compound request that defines a grouped graphical object The 
object once defined can be custom moved anywhere on the screen or resized. The 
CREATEVISUAL, SHOWVISUAL, HIDEVISUAL, l\/IOVEVISUAL and RESIZEVISUAL requests are 
intended to support this. The Visual Object is intended to support icons and graphical cursors 

30 typically with Alpha Blending. A future version of this protocol will address even more complicated 
graphical operations using Visual Objects. The PAUSE operation can be used along with 
MOVEVISUAL operations to indicate cursor movements. Typically, it is required that small 
sequences with very short pause times are specified. This is because of the intention to avoid 
excessive threading requirements on the client device, or handset In this version of the protocol. 

35 Compound Operations are also used by the input system to indicate event sequences. 

1 . 1 .2 Overview of the Event Handling Protocol 

The Event Handling Protocol defines the actions taken by the event system. The actions of 
the VP Client are ouflih^ in t^ Slow Se W 
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Chance to handle requests sent to It The goal of the VP protocol In this Is to allow the handset a 
simple way of sending user events baci< to servers and the application to handle them. The vartous 
events that are defined in tiie protocol represent actions by the user or are the result of actions on 
the system by the user. 

5 Typical actions include Touch Events, Key Events, Mouse Events and Speech Events. There 

are sequence numl)ere that typically get reset across, active sessions. The events are delivered in 
order and processed in order, VP Clients also deliver the causal events that let the state of the 
system change. 

The event handling protocol that the client implementation must follow requires that evente be 
10 delivered in order and that duplicate handling of evente does not tei<e place. For tills sequence 
numbers are used. This must proceed in sequence from 0 to 65535 and ttien rotate back to 0 on 
any given session. The connection and stete on tiie client for ttie Event Handling Session must 
follow the same stetes as ttie Drawing Protocol. This one-tOK)ne association guarantees ttiat evente 
are not delivered incorrectly. Thus when VP Ciiente transition fliey must transition botti connections. 
15 The Event Handling protocol is subsequenfly tied to the handling of various simple actions. 

At the time of EventSystem Initialization, ttie VP Client specifies tiie various event systems 
resident on the device. If Speech (input) is one of tiiem, then tiiere could be multiple possibilities 
about how tiie server could get speech input from tiie client and additional session level Interactions 
might need to be established to enable tills. 
20 It is additionally possible for ttie VP Client to use a proxy or an agency to deliver events to the 

VP senrer. The presence of a proxy could simplify interactions by allowing VP Clients to persist witti 
connections, allow tiie VP Client to send speech to a different location from tiie ottier evente or to 
allow Input to be sent to multiple applications witiiout much reconnect time. 

25 1 .1.2.1 Mouse, key and Touch Panel Evente 

Mouse. Key (or Bution) and Touch panel evente generate requeste ttiat get sent to ttie senrer. 
The evente are delivered in sequence over the wire. These are baditional Input device evente tiiat 
specify a click or a pr^ or a touch panel touch. Typically ttiey specify a location. In ttie case of Key 
Panels, tiie DTMF or keypad touch is delivered bansiated as an ASCII character string. Key panel 
press evente should be batched and sent as KeyStiing Event requeste as much as possible. They 
can also be echoed on flie local temnlhal when ttiere is local support for Widgete. (for instance, Text 
Boxes. TextAreas and TexH^ields will echo tocal evente). Typically, more sophisticated devices witii 
a small i<eyboard should tiy to support TextAreas and TextRelds to allow simpte and typing 
functions, (but tills is only a recommendation). If ttiey do not support ttiese Widgete, fliey should try 
to support ttie DrawRectTextBox function. When DrawRecfTextBox Is supported, ttie handset 
should echo ail data upto tfie point where ttie TextBox is full leaving additional actions to be taken 
by ttie sen/er. Graphiti writing as seen In contemporary dient devices also tell under flie purview of 
Key evente. 
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If none of TextField, textArea Widgets or TextBoxes is supported, the Event System stiould fall 
bacl< to echoing characters back to the server. 

Touch Panel Evente too can be compounded as a string of actions and sent to the server. 
These are line drawings, etc. Touch Panel Events can also fom the basis for actions to a 
5 handwriting recognizer (application). In this case, the sequence of events is sent to the server as a 
- compound event using DELIVEREVENTS. The Server then^ recognizes the character and issues a 
DrawChar or DrawString Event 

1 . 1 2.2 State Transition Events 

10 The various state transition events that take place with respect to session connections are 

also sent back to the Sender (Application). Servers too, can initiate state transitions through the 
CHANGESERVER Reque8t_(Called an AppHcation.Worm Hole.jthis is similar to a hyperlink button, 
and causes a new application to be launched (Change Server results in the VP Client reconnecting 
with a new server which could be on either the same machine or on a different machine). The Client 

15 could also initiate a link to a new Application from the client device. Subsequent to the request to 
launch the new application (on another server) the server sends the CHANGESERVER request. 
The VP Client could check with the user at this point or could go ahead and move the connection 
into the SUSPENDED state (as recommended by the CHANGESERVER REQUEST) or the 
TERMINATED STATE if the Application is Exiting. State Transition Events can also occur from 

20 User Actions that are local to the Client. The result is that the Client moves to a new state with 

respect to this application. State Transition Events can be delivered back to the Application Itself. It 
is typical although not necessary for servers to suspend applications when the client is moved to the 
SUSPENDED state. 

Typically, state transition events are sent to indicate a change with respect to a session with 
25 an application. It is possible for the VP Client to seek the user's advice on whether to disconnect or 
terminate the cunent set of active applications at the time of powerofT. (not necessary). 

A system list of the current set of applications running on behalf of the users can be obteined 
using the APPLICATIONLIST request This request Is made to the ApplicationListManager (ALM). 
The VP Client can use this infomnation to browse and decide the list of Applications the User is 
30 currently mnning and where (and additional information about them). This gives the user access to 
his environment and the ability to reconnect to any application in it 

1 .1 .2.3 Speech Interactions 

The role of speech is to enable speech and multimodal applications and act as a means for 

35 delivering it over a shared channel. Speech can be both an input and output In such applications 
and the event system deals with the use of speech as Input Speech is sent in one of several 
formats to the server. Speech Recognition systems on the server would recognize the spoken word 
and the application would display and verify what was spoken. This acts as a feedback mechanism 

^to the user-and helps correct and rectify ennorsrSpeech ran also be~a means to venfy and 
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authenticate a user and speaker identify systems have been in use. It Is preferable to use Raw 
Speech for speaker identify. 

1 .1 .3 Overview of tiie Speech Interactions and Protocol 

Speech can occur both in tiie data channel or outside flie data channel, (over a separate 
channel). In GPRS Systems, ttie system might support sImuKaneous modes as well as a swap mode' 
or only support data moda Multimodal interaction is posslbte In all tiiree modes. Typical signal 
processing systems use Voice Activity Detection (VAD) to detect momente of silences and these are 
used in contemporary cellular voice systems. Multimodal Man-Machine Applications Involve much 
longer silences tiian regular phone converations and involve much less telk typically. There is a 
burst followed by ottier actions and tiien a burst of speech. Afthough separate votee channels can 
be established and used ttirough ti» lengUi of ttie appllcation, such usage may not seem efficient 
for multimodal applications. The otiier possibility is ttirough tiie date channel. When speech is sent 
over the data channel, ttie VP Client could choose to etther send it as part of the DeliverNextEvent 
RPC request or using a separate channel or IP connection using ttie Session Initiation Pmtoctil and 
RTP/RTSP to send data to the sen/er application. It Is also not necessary for ttie speech date rate 
in multimodal applications to match tiie channel rate (it could be higher instantaneously) as one 
could assume tiiat even tiiough the data channel rate is low, the bursts of speech activity allow one 
to spread. tiie speech overa longer time. Furtiier in Man-Machine communications, delays are not 
as important as in person-person communications. The fomnats for speech are specified in ttie next 
section. Several formats are possible. The client device and tiie sender must agree on a format. It Is 
preferable If ttie speech recognizer and the handset agree on Oie fomiat tiiat tiie recognizer uses 
internally as a conversion from one fomiat to anottier might lose speech infomiation. This Is again 
decided as part of the capabilities and negotiations. 

If Speech is sent over ttie dedicated speech channel and involves a phone call ttiat temiinates 
just across ttie air Interface (probably at an MSG), tiien it is typical for speech to be recognized by a 
speech recognition system ttiere. This could be ttie same location as ttie application. If it is not, ttien 
speech could be fonvarded (relayed) to ttie actual sen/er using a sen«r-to-8erver Media Gateway 
Session (and is beyond ttie scope of ttiis specification). If Speech is sent over ttie date channel 
using ttie Event System Connection and as part of ttie DeliverNextEvent RPC, ttiere Is no need for a 
separate Speech Corinection. If ttils is found to be unsuitable by ttie client and sen/er, ttien It is 
possible to have a separate RTP/RTSP connection and ttiis is negotiated as part of tiie Event 
System Initialization. It Is also posslbte for ttie VP Clients speech to always be sent ttirough a proxy 
to ttie current Application Sen/er. It is also important to note ttiat Speech efther recognized or 
unrecognized can be forwarded back to the Application List Manager Ul (and is discussed teter). 

Speech can be sent in various fonns. There Is ttie simpte form of speech ttiat Invoh/es 
sampled raw speech. Anottier form could be ADPCM. A tiiird form of speech is ttie one used in 
GSM Networks (RPE-LTP). 3G PP has specified AMR which has many internal modes of ACELP. A 
fiftti fomiat is ttie one used by flie Sphinx systemlflmicfi i^^^ 
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involves oepstral and delta cepsbal parameter transmission, (input from Companies is needed for 
this) It is prefeiabie if some fonn of noise cancellation is done as part of the process. This could be 
done either at the sen/er or at the client 

1.1.3.1 Input Methods and Speech Recognition 

- The several (pointing device )input methods used like Touch Panel. Mouse, etc. can be 
complemented by speech. Speech input can be fed back into the user interlace. For Instance, a 
user might specify the new cursor setting using the Touch Panel and then use speech input to 
update the TextFleld he has targeted. Also it is possible for speech inputs to cause multiple actions 
both in the active APPLICATION or in a secondary application. 

Speech Input can also serve to enable meta-actions on th^ user interface. For instance, a 
"List file menu" request might imply that the Ul list the File. menu. Such Interfaces can be developed 
for an application. Speech can also be input to a VoiceXML browser. Several additional possibilities 
exist 

1.1.3.2 Speech Output Methods and Text-to-Speech Conversion 

The Speech Aspects of the protocol relate to delivery of Text for TexttoSpeech conversion 
when the system supports it and the delivery of raw speech (in the negotiated fbmiat) to aid Speech 
output to support Multimodal applications. 

1 .1 .3.3 Recommendations for Multimodal Applications aiid Server Behavior 

Multimodal Applications will havea significant Impact on tiie future of mobile computing. 
Multimodal applications require that ttie platform and ttie application interoperate to provide a unified 
look and fsel Involving both an audio and a visual inpul/output mechanism. Multimodal Applications 
accept Input from key and touch panels as well as from Speech. The Spoken word could lead to a 
meta-action (like help on ttie overall screen), might be context sensitive to tiie current location of the 
cursor or might lead to ttie launch of a new application (and have been intended for the Application 
Manager). The semantic import of ttie speech couW ttius vary wltti ttie User Intertece. The use of 
deictic gestures (Including pars-pro^ delxis) wltti coordinated speech input oflters new and 
powerful user interfaces tiiat couW easily for tiie basis for building compelling and attractive user 
interfaces. The interactions to support ttiis might Involve additional application level protocols, server 
to server protocols and use conditions tiiat go beyond ttie scope of ttiis specification. Furttier, 
multimodality brings added robustness as enrors in speech recognition could be recHfied ttirough 
visual reverification. Typical Multimodal Applications include VoiceXML browsers, futojre Multimodal 
Markup Language browsers, java.speech enabled AWT applications, etc. The W3C is actively 
studying ttie scope of multimodal web interfaces and is likely to also create a mariaip language to 
bring new killer applications[15]. 

-1 .-1.4 Overview-of Multimedia-interactions and-Protocol 
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The main multimecfia interactions relate to tlie delivery of audio and video l)etween a VP 
Server and the VP Client IMuitimedia Audio and Video are heavy bandwidth users and need 
realtime support This makes multimedia streaming a challenge in mobile networks. Heavy rate 
control and error robustness handling Is required for realtime streaming in Wide Area Wireless 
environments. Instantaneous rates can both exceed the channel rate as well as the random moving 
average rate from the original source can be too high for use on the particular channel. This Implies 
that the VP Sender must reduce rates to match the available bandwidth In the air interface through 
multiple means including k)ssy compression schemes, recompression, lowering screen rates, etc. 
Users should also have a say In deciding the rata they want and they might want higher rales for 
certain kinds of activity and lower rates for others. 

The MExE stage 1 specification require fine grain user control over streamed media. 
Currently, 3G PP is specifying a Packet Switched Streaming Service (PSS [16]) for media 
streaming. This specifies multiple formats including MPEG-4 and H.263. In addition, a multimedia 
messaging service is being defined by 3G PP to support the delivery of multimedia messages to 
clients. This too defines several formats and protocols to be implemented by the MMS User Agent 
Various capabilities provided in MPEG-4 are again optional features. 
• VP Cliente can have IMultimedia support as a strictly optional feature. 

1 . 1 .5 The Mobile File Store Interactions and Protocol 

At the broadest level, user state typically includes pemnanent storage including owned 
application storage, data and applicatons. 

Permanent storage is scarce in most cunnent day handsets and handheld computers. Thus 
pemfianent storage inside the network seems a viable option for end users. VP Clients could support 
this through several means. The first would be to implement a networked file system to browse data 
fiTom handsels. NFS V4 is also built around ONC RPC and shares several features and functionality 
with VP Clients. This could be used to implement a nelworiced file access system. Another 
alternative is for the Application Manager to provide a visual file browsing capability from the server 
to a virtual view of the users files located inskle the network. 

Server supported file set migratfon and replication capabilities are additionally important in 
mobile applicatton environments and the presence of a mobile fite store and the ability of users to 
manage it might serve as a complementary addition and support the notion of a Virtual Home 
Environment that users find ubiquitously avaitable. The Mobite Rte Store is a optional and 
oonnplementary. 

1. 1 .6 Security Interactions and Protocols 

The VP Client protocol uses a slight variant of the RPC specification (and XDR) defined in 
RC1831 and RFC1832. Several security flavors traditionally existed for RPC notebty Unix Security. 
DES. Kertseros. In recent years, RPCSEC_GSS security flavor as defined In RFC2203 (and uses 
-GSS-API as specified in RFC2078) has been proposed for the addlHoh of stronger- and more secure 
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Wide area connectivity. RFC2847 specifies the LIPKEY Inftastructure for use with RPC. This is a low 
infrastructure public key system for use with RPC. 

Further discussion on the detailed methods for implementing Security on a VP Client is 
expanded in brief in Section 4. Detailed specification is defenBd to a sister document of this 
specification. 

1.2 General Definitions 

The definitions below are used to illustrate the use of temf)lnology In this document Additional 
temiinology reference is also available from 22.057 the MExE Stage 1 specification and 23.057, the 
MExE Stage 2 specification and a few referenced documents. This draft vereion does not however 
predefine all ternis prior to use and several must be understood in the context of their use. 

The term "VP client" describes the software on the User's Equipment which 
handles the display and input of the client device and implements the VP 
Protocols and interactions described in this document 

VP Server A server that runs applications on behalf of the VP Client and uses the VP 

Protocols to interact with the Client 

Bootstrap Server Runs DHCP and enables the VP Client to find a variety of infomiatlon that it is 
looking for. Also runs the Application List Manager 

The Application List Manager is a service that runs on the Bootstrap Server and 
Implements the server side of the Application List Protocol. At the VP Client 
request it could launch a new application or connect the user to application to 
any of his existing applications or launch the Application Manager Ul 

The Application Instance Is an executing VP Application that is launched and 
running on a VP Sen/er. 

1 .3 Overview of the Interactions for bootstrapping VP Clients 

This section is only a recommended guideline for bootstrap in the context of IP. VP Clients 
could use alternative mechanisms to bootstrap themselves and VP Clients could mn on alternate 
transports and networks. 

A variety of interactions exist for bootstrapping VP Clients. They include the initial DHCP 
request to the DHCP Bootstrap Server. Typically every handset should be in an environment that 
has a DHCP server. Using the DHCP responseland fo^ 
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Client establishes a connection to the Application Manager and could then either automatically 
return to the previously running Application (by querying the Application List Manager) or launch the 
Application Manager Ul. From the VP Clienf s perspective the core of the bootstrap process Is 
intended to return to the previously existing state of the VP Client. 

. . .1 .3.1 DHCP and bootstrap support in the Client device 

The DHCP bootstrap process is intended to enable VP Clients to gain an identity (like a 
mobile IP address) and connect to their existing state In any environment that the user is In. DHCP 
is defined In RFC2131. DHCP extensions to support VP Clients in Mobile IP environments vinll be 
defined in a related sister specification. 

A variety of servers and services information is needed as part of the VP Client bootstrap 
process. These include security and authentiratiqn services.and inlbrmation. User Identity - - - - - 
information, and other configuration information. Additional Host Profile identification and 
information could be obtained through server-to-sen^er protocols and is beyond the scope of this 
specification. 

1.3.2 The Application List Manager 

The Application List Manager (ALM) is an RPC service that runs on a server identified by the 
DHCP server in the client device's service environment It is intended to support the client device's 
environment and its mobility, it is implicit that the Application List Manager in a mobile environment - 
will involve a server-to-server protocol or mechanism to obtain infomiation on the user's cunent 
environment and how to connect bacl< to it. 

The ALM also helps the client device join one of its existing applications or binds to the last 
nin application. 

The ALM and Its associated protocol is designed to offer a client device and the MExE service 
environment a supporting framework to enable the Virtual Home Environment or VHEpj. The 
intention is to allow serving networks to provide complete access to a user's environment his 
njnning applications and to enable usere to personalize their sen/ice environments. The ALM 
mechanisms are Intended to use existing HLR and VLR databases vyith extensions to handle and 
support the access to and the easy provisioning of new senrtces. The retrieval and reestablishment 
of previously established state is a cornerstone of VP and a necessary requirement for the Mea of 
ubiquitous computing. 

1.3.3 The Application List Manager Protocol 

The Application List Manager Protocol is used to communicate with the ALM. A VP Client 
sends ALM requests to the ALM RPC sen/ice. The protocol senses to retrieve Applfeation 
information and for Application Discovery and connection. Itisa mechanism to enable VP Clients to 
find appropriate VP Servers. ALM Servers cooperate with each other to discover the user's existing 
applications and application environment - - - 
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New Application Launching is handled in cooperation with the AIM. VP Clients being mobile 
migrate to new virtual environments. In these environments, new applications could nin on 
appropriate servers and new applications might exist The AIM is intended to support this 
mechanism. The VP Client uses the ALI^A Protocol for its Application needs. VP Servers also use 
the AIM protocol when there Is a button or link that leads to the launch of a new application, in this 
. case, the ALM Protocol as well as additional mechanisrns could be used to obtain an appropriate - 
Sen/er. Once the appropriate server Is found, the application is launched on this server and the VP 
Client is informed about it 

Some of the server- to -server aspects of AI.M is beyond the scope of this specification. 

2. Protocol Data Types 

This section provides the various data types used by the protocols listed in this document The 
various data types listed here are used to implement the protocols required. 



15 2.1 Basic Data Types 

Data Type Definition 



int32 t 



typedef int int32J; 



20 uint32 t 



typedef unsigned int uint32_t 



int64 t 



typedef hyper int64_t; 



25 



uint64_t 
shortj 



typedef unsigned hyper uint64 J; 
typedef short ushortj 



ushort t 



typedef unsigned shortj; 



30 Uchar 



typedef unsigned char Uchan 



Uchararray 



typedef unsigned char Ucharanayo; 



35 



Fid typedef short: Fid; 

Namestr typedef string Namestr<MAXlSlAMELEN>; 



Drawable 



typedef opaque Drawable[DRLENGTH]; 
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Parray 



fypedefintParrayo; 
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SeqJ • typedef unsigned short seqj; 

Wid lypedef short Wid; 

E"""® fypedef unsigned short enums; 

Procnums typedef unsigned short Procnums; 

VPstat typedef unsigned short VPstat; 

Reqnum lypdef unsigned short ReqNum; 



2.2 Constants and Enumerations 

const RGB = 3; 

const RGBA = 4; 

oonstYUV = 3; 

const MAXNAI\/IELEN = 255; 

const StaticGray = 0; 

oonstGrayScaiesi; 

constStaticCoiors2; 
const Pseudocolor = 3; 
constTrueColor=4; 

constDRLENGTH = 4; 

const CURRENTDRAWABLE = 0 

const SAVEDDRAWABLE = -1; 

enums sessionstate { 
ACTIVE = 0, 
SUSPENDED =1, 
DISCONNECTED = 2, 
TERI\fllNATED = 3, 

}: 



enums colors/stem { 
CL^RGB565= 1, 
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CL_RGB888 = 2, 
CL_RGBA8888b 3, 
CL_YUV888 = 4 

}: 

5 , • 

. enurns eventsystem { - 
EVT_M0USE = 1, 
E\n'_REMOTE = 2, 
EVT_KBD = 3. 
10 EVT_JOYSTICK = 4, 

EVT_SPEECH 

}; 

const MAXEVENTSYSTEMS = 8; 

15 

enum imgtype { 
JPEG = 1, 
GIF = 2, 
PIXMAP = 3 

20 }: 

enums frngjenum { 
RETOK = 0, 
RETRY sl. 

25 N0MEM0RY = 2 

}; 

const MAXIMAGEDATA = 8192; 

30 enums mask { 

VisualNoMask = 0, 

VisuallDMask= 1, 

VisualScreenMask = 2, 

VisualDepthMask =4, 
35 VisualClassMask = 8, 

VisualRedMaskMask = 16, 

VisualGreenMaskMask= 32, 

VisualBlueMaskMask = 64, 

VisualGolonTiapSizeMask=-128, 
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VisuaiBftsPerRGBMask = 256 

}: 

enums Fnttype { 
THIN = 0x1, 
LIGHT = 0x2. . . 
REGULAR = 0x4. 
BOLD = 0x8, 
ISITALIC = 0x16, 
ISUNDERLINE = 0x32. 



enums Labelhow { 
LEFr = 0. 
RIGHT =1, 
CENTER = 2 



enums ScrollbarType { 
VERTICAL = 0. 
HORIZONTAL =1 



enums TatBaScrollbars { 
. SCROLLBARS_BOTH. 
SCROLLBARS_NONE. 
SCROLLBARS_HORIZONTAL_ONLY. 
SCROLLBARS_VERTICAL_ONLY 



enums SetTextHow { 
START = 0, 
APPEND = 1, 
INSERT = 2, 
REPLACERANGE = 3 

}; 

enums MenultemType { 
MENUITEM = 0. 
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SUBMENU si, 
CHECKBOXMENUITEM »2 

}: 

5 enum8 0nOpType{ 

. ONCUCK=0, - 

ONOVERsi, 

0N0UT=2, 

ONCLICK2S3, 
10 0NSELECT=4, 

OnScrollLUP =5, 

OnScrollLDOWN = 6, 
' ' bnScrollPUP = 77 

OnScrollPDOWN = 8 

15 }; 

2.3 Structured Data Types 
struct coli^btype { 

20 r 

byte n 
byteg; 
byteb; 

*/ 

25 charrgb[RGB]; 

}; 

struct colrgbatype{ 
/• 

30 byten . 

byteg; 
byteb; 
byte a; 

*/ 

35 charrgba[RGBA]; 

}: 

struct colyuvtype { 

/S : 
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bytey; 
byteu; 
bytev; 

*/ 

charyuv[YUVJ; 

}: _ 

struct colothertype { 

opaque colcookiep]; 

opaque coival<COLSIZE>; 

); 

constDRLENGTH = 4; 



enums colorsystem { 

CL_RGB565= 1, 
CL_RGB888= 2, 
CL_RGBA8888 = 3, 
CL_YUV888 * 4. 
CL_GRAY2 = 6, 
CL_GRAY4 = 6, 
CL_GRAY8 = 7, 
OTHER = 8 

>; 

enums eventsystem { 
EVT.M0USE = 1. 
EVT.T0UCH = 2. 

EVT_KBD=3. 

EVT_REM0TE = 4, 
EVT_SPEECH=5, 
EVT_J0YSTICK = 6 

}: 

const MAXEVENTSYSTEMS = 8; 
constCOLSIZE = 8; 
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union Color sivltch (colorsystem colsys) { 
caseCL_RGB565: 

ool565type colrgb565; 
caseCL_RGB888: 

oolrgbtype oolrgb; 

.<aseCL^RGBA8888: - . - 

coligbalype colrgba; 
caseCL_YUV888: 

colyuvtype colyuv; 
case CL_GRAY2, CL_GRAY4, CL_GRAY8: 

char oolgray; 
case OTHER: 

colothertype colval; 
default 

void; 

): 

union ColAnray switch (colorsystem colsys) { 
case CL_RGB565: 

col565type igb665array<>; 
ca8eCL_RGB888: 

colrgbtype fgb888an3y<>;. 
ca8eCL_RGBA8888: 

colipbatype rgba8888array<>; 
case CL_YUV888: 

colyuvtype yuv888anay<>; 
case CL_GRAY2. CL_GRAY4, CL_GRAY8: 

char colgrayanayo; 
case OTHER: 

colothertype colvalarrayo; 
defiault 

void; 

}: 

struct ScreenParams{ 
int ScreenID; 
lot ScrWidth; 
int ScrHelght; 

Int- ScrWIdthMM; -■ ■ 
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Int ScrHeightMM; 
color^tem ScrCol; 
OSC_resp oscresp; 
DrwRespRootOrawable; 
maybe we need a Font Here *l 
}; - - - - 

enum imgtype { 
JPEG = 1, 
GIF =2, 
PIXMAP = 3 

Y_ ; 

enum img_enum { 
RETOK = 0, 
RETRY =1, 
NOMEMORYs2 

}; 

const MAXIMAGEDATA = 8192; 

union ImgAnay switch (imgtype itype) { 
case JPEG: 

opaque JPegData<MAXIIVIAGEDATA>; 
case GIF: 

opaque GifData<WAXIMAGEDATA>; 
case PIXMAP: 

ColAnrayColorArn 

}; 

struct EventPaiams { 

Int EventSysID; 
eventsystem evttype; 
boof DragAndDrap; 

}; 

#defineMAXRECTS1 
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staict Rectangle { 
IntX; 
IntY; 
int Width; 
int Height; 

}; - - - 

typedef strudt Rectangle Rectangle; 

stnjct RectSize { 
short Width: 
short Height; 

' " " ' 

struct Coordinate { 
short X; 
short Y; 

}: 

union WidgetU switch (bool created) { 
case TRUE: 
Wid widget; . 
case FALSE: 
void; 

}; 

union WidgetLlstU 

switeh (boo! created) { 
case TRUE: 

Wid widgeto; 
case FALSE: 

void; 

}; 

Union WIdsize switch (bools recommend) { 
Case TRUE: 

RectSize size; 
Case FALSE: 
^Vold; : 
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}: 

struct Button { 
Label label; 

}; 

stmctCreateLabelReq { 
int ScreenID; 
Drawable drw; 
Button button; 

}; 

struct Checkbox { 
Label label; 
boo! on; 

WidU checkboxgroup; 

}; 

stmct Cholceltem { 
Uarray Name; 

}; 

struct errseqreply{ 

SeqJ received; 
SecLt expected; 

}: 

union ObjectU switch (bool created) { 
case TRUE: 
^ Objid Object; 
case FALSE: 
void; 



enums speechformat { 

SPEECH_CALL=0. 

SPEECH_SIP^RAW = 1. 

SPEECH_SIP__RPE_LTP = 2, 
...>_._SPEECH^SIP.SPHINX = 3, 
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SPEECH_EVT_RAW=4, 
SPEECH_EVT_RPE_LTP = 5. 
SPEECH_EVT_SPHINX = 6 

}; 

5 



2.4 Error Definitions 

VPOK- Request was perfomied successfully 

10 

VPCOMPNOTSUPP - Compound Operations is not supported. 



VPONOPSNOTSUPP - Does not support "onxxx" operations 
VPUNWRAP - unwrap the compound operation as it is too complicated 
VPNOTSUPP - Procedure Is not supported 

STALEDRAWABLE - The drawable supplied is stale - (recreate and replay the sequence on 
ttie drawable to reestablish its state). 

OUTOFSEQ - Sequence Number failure - (received, expected seqnum is appended) 
VPINVAL - The argumentsupplied was invalid 
VPCOLSYSNOTSUPP -The ColorSystem chosen is not supported 
EVENTSYSNOTSUP - One of the EventSyslems chosen is not supported 
ll\/iAGESYSNOTSUP - One of the ImageSyslems chosen is not supported 
FONTSYSNOTSUPP - one of the FontSystems choseri is not supported 



35 3. RPC/XDR Usage and Behavior 

VP is mostly based on the use of RPC and XDR[6]. [7] and is intended to completely reuse 
existing specifications, leverage security styles and because of the ubiquitous avaiiablli^ of both 
software and documentation on the subject 
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3.1 Deviations from the RPC/XDR specification 

One departure from RFC1832 and RFC1831 is that data in the XDR fomiat is rounded off on 
2 byte boundaries rather than 4 byte boundaries. This is due to the need to oonsen/e bandwidth and 
is one of the methods to reduce bandwidth usage. 

A further optimization is the definition of a short enumeration enums and a short boolean type 
..bppte. . .... - - - 

3.2 Sequence Numbers and their use 

Sequence Numbers are used to Identity requests and most protocols require actions to be 
executed In order. Thus every RPC request in tfie VP protocol (and the Event System Protocol) 
begin witti a sequence number. TTiis is implied in the sections 5, 6 and 7 protocol specifications. 

4. Security Interactions and RPCSEC_GSS 

Security Interactions could change based on Vrte location of usage, it is Important to note ttiat 
VP Clients can be used in both local environments, virtiere ttie level of tiiistand security is higher 
and hence does not require sophisticated security, as well as wide area environments where 
sophisticated security will be very important For this reason, some of ttie implications of security 
must be carefully explored in the context of mobile environments. 

The VP protocol uses RPCSEC_GSS where appropriate and leaves ttie specifications for 
security to a related specification witti the important comment ttiat much of ttie onus of supporting a 
variety of flavors is implicitly placed on servers. VP Clients could choose to implement just qne 
appropriate security flavor ottier ttian AUTH^NONE. 

6. The Graphics and Speech Interiiace Protocol 

The detailed specification of tiie Graphics and Speech Interiiace Protocol is outiined below. 
The protocol is designed to enable VP Clients to service requests from a Server. It defines ttie 
protocol interactions and behavior of ttie dlent . 

Please note ttiat tiie terminology of Client and server used here is ttie reverse of traditional use in 
RPC environments! The VP Client is the server side of ttie RPCs 

sent by ttie VP Server. To be consistent ttiis document uses VP Client even when ttie software 
described is ttie server side of RPC. 

5.1 Null Operation 

Procedure 0: NULL- No Operation 

SYNOPSIS 
<null> 

—ARGUMENT 
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Void; 

RESULT 
Void; 

5 

DESCRIPTION ' 

Standard NULL procedure. This is used to measure overhead or is sometimes overloaded to 
handle initial securify negotiations. 

10 ERRORS 
None 



5,2 Compound Operations 

Compound operations is adapted from its use in NFS V4. Although compound request 

15 support is optional on clients, it can be said that It is a recommendation for handsets with more 

advanced capabilities, particularly those targeted for Wide Area low latency networks. The model for 
compound operations is for the client to service requests in order till a request fails. On failure, 
subsequent operations are not handled. The operations happen on the "cun-enf drawable. There is 
also the "saved" drawable. 

20 The basics of the COMPOUND request involve sending a series of (op + args) requests. 

The reply involves a con^esponding series of status and results. The onus is on the server to handle 
recovery from emors. (if it keeps track of requests it can choose to reestablish state on the drawable 
or reshape the Ul). 

25 SYNOPSIS 

. Compoundargs □ compoundres 

ARGUMENT 

30 

union VPjargs switch (procnums proc) { 
Case <OPCODE>: <argument>; 

} 

35 . _ 

struct Compoundargs { 
VPjargs VP_arganay<>; 

^ L. „ 
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RESULT 

Union VP_res switch (prpcnums prop) { 

5 Case<OPCODE>: <afgument>; 

} 

struct Compoundres { 
10 VPstat status; 

VP_res VP_rearray<>; 

}: 

DESCRIPTION 

15 The Compound Procedure combines multiple VP requests into one single request The VP ' 

client perfbmis the individual requests of the compound request in order till one of the requests fail. 
The result Is the same as that of the last request executed. Several operations have a void response 
(these are mandatory requests that involve simple 
Drawings). No bytes are sent in the response for these. 

20 

IMPLEMENTATION 

The Implementation requires that the VP Client confonn to the capabilily negoliatfon agreed 
upon. VP Clients that support Compound requests and the OnAcHon operations defined would need 
25 to cache compound requests on the root drawable. 

ERRORS 

All defined errors. 

VPCOMPNOTSUPP, VPONpPSNOTSUPP - can both be returned by Compound. 
30 In addition, the client can either say that it does not support compound operations or ask the 

sender to unwrap the compound operation as it is not able to execute the long sequence. It can also 
say that it does not support Onxxx operations. . 

5.3 Client deviceSystem Info 
35 SYNOPSIS 

VPSystemlnfbaigs □ VPSystemlnfbres 
ARGUMENT 
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Union SpeechSupport switch (boots supported) { 
case TRUE: 

SpeechFonnat speechfomiato; 
Case FALSE: 

Void; 

} " " - ^ - - - 

VPSystemlnfoargs { 
Bools supportcompound; 
Bools supportonop; 
SpeechSupport speechlnput; 

SpeechSupport speechoutput; 

colorsystem ooisyso; 
eventsysem evtsys <>; 
imgtype imgsyso; 
fontsystems fontsys<>; 

}; 

RESULT 

VPSystemlnfores { 
Bools supportcompound; 
Bools supportonop; 
SpeechSupport speechoutput; 
SpeechSupport speechinput; 
short NumActiveDrawables; 
short NumCachedDrawables; 
int64J SuppOptProcflags; 
short ScrWidth; 
short ScrtHelght; 
short ScrWldthlVII\^; 
short ScrHelghtMM; 
colorsystem ScrCol;. 
Bools supportfbrVisuals; 
Boois supportfbrfillstyles; 

eventeystem evisyso; 
imgtype imgsyso; 

fontsystems fontsyso; 

J 
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DESCRIPTION 

This represents the capability negotiation that takes place between the client and the server 
regarding the VP Clients display and Input capabilities. The,VP Server informs the VP client ... 
regarding its own support for the display size, color system used, the supported input and output 
capabilities, support for compound operations etc. It is advisable that VP Servers support the 
broadest possible set of capabilities. 

The client replies with indications about its own support for Compound Operations, OnAction 
operations, various forms of speech output etc. from the list of capabilities the server supports. This 
results in a lower common subset as the negotiated capabilities used by servers to access VP 
Clients^ ^ ^— - - - 

The reply also indicates if the VP Client supports colorsystems beyond the identified systems. 
If this is the case, the VP Server can then use GetVisuaJInfo to query for additional capabilities. 
This Is a mandatory procedure and all VP Clients must support it. 

IMPLEIWENTATION 

It is advisable that VP Clients look to follow recommended profiles as advised by the 
Classmark. This will ensure that servers support their capabilities. However, it is better if servers 
support the widest possible array of VP Client capabilities. 

A VP Server might not use this operation if it has previously negotiated and might choose to 
directly proceed with either initializing the graphics system or directly to request actions on 
drawables, caches etc. 

The recommendations that VP Clients make on Cached Drawables are purely advisory. As 
they disconnect and reconnect it is possible that they might choose to discard cached state. The 
Onus is on servers to always maintain state and be able to recover when VP Clients have discarded 
state across disconnections and reconnections. 

VP Client implementers must note that servers can use server to server protocols to transfer 
negotiated capabilities to minimize over the air traffic. 

ERRORS 

INVAL - if the arguments supplied by the sender are invalid 

5.4 InitializeGraphicsSystem 
SYNOPSIS 

InlBalizeGraphlcsSystemargs □ InitializeGraphicsSystemRes 

ARGUMENT 

Struct InltiaiizeGraphicsSystemargs { 
Colorsystem Scrcol; 

' " " ' ^ ' ' ^46- 
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Color fg; 
Color bg; 
short ScrWIdth; 
short ScrHeight; 

eventsystem evtsyso; 

. . imgtypeJmgsyso; 

fontsystems fbntsyso; 
boots isnewapp; 
bools IsRootActive; 

opaqueAppGoolde[APPCOOKIESIZEI; 

}; 

RESULT ' " ' • 
struct Offecreenresp { 

short NumDrawables; 

}; . 

union OSC^resp switch (bool hasoffscr) { 
case TRUE: 

Offscreenresp offscr. 
case FALSE: 

void; 

}: . 

union DrawableCache switeh (boo! supportcache) { 
case TRUE: 

Drawable drcacheo; 
Case FALSE: 
Void; 

} 

union DnA^Resp switch (bool created) { 
case TRUE: 

Drawable drvi^ 
case FALSE: 

bools updated; 

}; 



stnjct-lnitializeGraphicsSystenriresponse"{ 
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bool hascookid; 
DrauirableCache diawablecache; 
coloreystem ScrCol; 
OSC.resp oscresp; 
OrwRespRootDrawable; 



union InitializeGraphicsSystemres switch (VPstat stat) { 

10 case VPOK: 

InitializeGraphicsSysfemresponse igs; 
default: 
void; 

}: 

15 



DESCRIPTION ' 

This is the graphics system initialization that tal<es place after clients and servers negotiate. 
The system attempts to specify whether the application is old or new and then advises the VP Client 
about the screen parameters previously negotiated, the foreground color, the background color, the 
20 event ^sterns negotiated, etc. 

The VP Client Is advised oii the Application and whether this is a new application or an old 
one. If ft Is new. it is advised on the whether there have been any intervening updates to the screen 
while the user was disconnected (If there were then IsRootAcBve is f&lse).The AppCookie must be 
unkiue.on the server across reboots, etc, Thus the tuple, sender address and AppCookie represent a 
unkiue Wenf fier for the applicatkm from the VP Clients point of view. A VP client that supports 
caches can indicate that It has the Appcookie by setting hascookie to TRUE 

The VP Client If it caches drawables then Indicates the list of cached drawables for this 
applicatfon. If the RootDrawable was also cached, the Client updates the screen with the cached 
version of the RootDrawabte (only if IsRootAcHve is TRUE). VP Clients that do not support caches 
must still oonfbmi with the protocol and set supportcache as FALSE. 



25 



30 



IMPLEMENTATION 

VP Clients that support Caching must cache Appcookies and sen«r tuples as well as the 
current Root Drawables to effectively return to previously existing and cached state. They should 
35 cache drawabtes whole or not at all. 

ERRORS 

INVAL - The arguments supplied by the server are invalid 
VPCOLSYSNOTSUPP -The ColorSystem chosen is not supported ' 
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EVENTSYSNOTSUP - One of the EventSystems chosen is not supported 
IMAGESYSNOTSUP - One of the InrjageSystems chosen is not supported 
FONTSYSNOTSUPP - one of the FontSystems chosen is not supported 

5 The enors are unlilceiy but on seeing these, the VP server will attempt to renegotiate. 



5.5 InltialtzeEven^ystem 
SYNOPSIS 

10 InitializeEventSystemAi^s □ InitializeEventSystemRes 

ARGUMENT 

- union SpeechSysAction switch {SpeechiFormatH 

Case SPEECH_CALL: 
15 String PhoneNum; 

Case SPEECH_SIP_RAW, SPEECH_SiP_RPE_LTP. SPEECH_SIP_SPHiNX: 
Address SiPServer; 

Case SPEECH_EVT_RAW. SPEECH_EVT_RPE_LTP. SPEECH_EVT_SPHINX: 
Short hostcoolde; 

20 }: 

union EventSysAction switch (evttype) { 

Case EVT^KBD, EVTJilOUSE, EVT_TOUCH, EVT_REiV10TE. EVT_JOYSTICK: 

Short hostcooide; /* this is typically a port address on the sen/er */ 
Case EVT_SPEECH: 
25 SpeechSysAction speechacHon; 

}: 

struct InitiaiizeEventSystemAtgs { 
EventSysAction evttypeo; 

30 }; 

RESULT 

struct EventParams { 

short EventSysiD; 
35 eventsystem evttype; 

}: 



union EventU switch (bool evtreply) { - 
.Qaae_TRUE- 
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EventPaiams evtpanns; 
case FALSE: 
void; 

}; 

5 

union InitiallzeEventSystemRes swltoii (Vpstat stat) { 
case VPOK: 

EventU evtso; 
Default 
10 Void; 
} 



DESCfRIPTION " 

This initializes the EventSystems for sending input back to the sen/er. The effect of this Is to 

15 initialize the list of event systems the VP Client supports and to connect to Speech systems when 
they are supported. This requires the initialization of connections by the Event System Protocol and 
the establishment of a conduit back to the VP sen/er from it The hostcookie Is typically a port on the 
server that the Event System Protocol could use (either directiy or through an agency). The request 
specifies the various possible event systems to initialize. The response however specifies the list of 

20 Event Systems that were actually initialized. 

If Speech input is supported the. PhoneNum or SIPAddress is provided by the server as a 
recommendation. 

Support for this request is mandatory. 

25 IMPLEMENTATION 

When Caches are kriovwi to exist and compound requests are supported, this request itself 
can be compounded with InitializeGraphlcsSystem. This will alkiw fewer over the alrtiansfers as 
part of reinitialization or initialization. 

In single threaded VP Clients this might take a period of time as the Event System is Initialized 
30 by the VP Client implementation, 

VP Clients could choose to switch io the PhoneNum or they could continue with tiieir existing 
number. If they do continue then the VP Client must Indicate back to the server on tiie 
InitEventSystem request as part of the initializafion of the Event System Protocol the sender that It is 
using. If it uses an agency or proxy for its regular event system requests, that too is indicated by the 
InitEventSystem request, (agency support will be explained by an addendum or reference). 

ERRORS 

INVAL -when arguments are invalid 
EVENTSVSNOTSUPP. vWien a system recommended by the server Is nofsup^^^ 



35 
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EVENTSYSCONNECTERROR - there was a problem connecOng or establishing the 
connection with a certain eventsystem. 

5.6 FreeGraphicsSystem 
5 SYNOPSIS 

. _ _ . FreeGraphicsSystemAigs □ Void - - - - - - 

ARGUMENT 

FreeGraphicsSystemArgs { 

opaqueAppCoofde[APPCOOKIESIZE]; 

}; 



10 



IS 



RESULT 
Void; 

DESCRIPTION 

Free the GraphicsSystem associated with the AppCookie. This frees all resources, caches, 
etc and disconnects the eventsystem as well 

20 IMPLEMENTATION 

Typically VP Cliente could choose to return to the ALM when this request is seen. This will 
allow the VP Clients to bind to additional applications or advise the user that the application has 
exited. 

25 ERRORS 

5.7 GefVisuallnfo 
SYNOPSIS 

30 C^tVisuallnfoargs □ GefVisuallnfores 

ARGUMENT 

Struct GetVisuallnlbargs { 
35 masl< masks; 

Vtemplate vtemplate< >; 

} 

RESULT : 
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Enums Colorclass { 

D1RECTCOLOR = 0. 
GRAYSCALE = 1. 
PSEUDOCOLOR = 2, 
5 STATICCOLOR = 3, 

STATICGRAY = 4, 
TRUEC0L0R = 5 

}; 

struct Visuallnfo{ 
10 . short depth; 

enums Colordass; 

unsigned long red_nissk; 

unsigned long green_mask; 

unsigned long blue_mask; 
15 short colomfiap_size; 

short bits_per_rgb; 

bools tilesandstipples; 

) 

union GetVisuallnfores switch (VPstat ret) { 
20 caseVPOK: 

Visuallnfo Vinfo<>; 

default 

void; 

); 



25 



30 



35 



DESCRIPTION 

For advanced Graphics Systems like gaming systems. GetVlsuallnfb allows the use of new 
colomiap handling and visuals. This concept Is adopted from X Windows. When the colorsyslem fs 
OTHER, this is used to obtain infomiatton about that oolorsystem. 

IMPLEMENTATION 

The use of this capability is mainly intended for supporting better graphics and can be specific 
to certain applications and devtoes. 

Many VP clients may not support this capabllHy. (depends on input). 

ERRORS 



VPINVAL - parameters are invalid 
VPNOTSUPP - this procedure is not supported 
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5.8 SetFillStyle 
SYNOPSIS 

SetFillStyleArgs □ VPstat 



ARGUMENT 

Enums FillStyle { 
FILLSOLID = 0, 
RLLTILED = 1, 
FILLSTIPPLED = 2, 
- FILL0PAQUESTIPPLED = 3 

}; 

union SetFiliStyleU switch (enums FillS^le) { 
case FILLSOLID: 

void; 
case FILLTILED: 

Pbcmaptile; 
Case FILLSTIPPLED: 

Pixmapstipple; 

} ■ 

struct SetFlllStyleAigs { 
Diawabte drw, 
SetFiliStyleU sfsu; 
Coordinate tsorigin; 

} 

RESULT 

VPstat 
DESCRIPTION 

SetFillStyle is again used in advanced graphics systems to paint a brush pattern while 
drawing or coloring. This request is used to develop better user interlaces in modem Ui systems. 
PIxmap represents the pattern used to transcribe: 
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IMPLEMENTATION 

SeJFIIIStyle is used to develop an Improved graphics system and can be used also in gaming 
etc VP Clients must support botii tiling and stippling. It can be used wltti compound operaflons to 
Implement interesting features. Thus advanced VP Clients oould support this capability notably if 
they are jntecestedJn gaming ete. 

ERRORS 

VPINVAL - arguments are invalid 
VPNOTSUPP - the procedure is not implenjented 
VPINVALDRAWABLE - the drawable is Invalid 



5.9 CreateDrawabie 
SYNOPSIS 

CreateDrawableAt^s □ CreateDrawabieRes 

ARGUMENT 

staict CreateOrawableReq { 
short X; 
shortY; 
short Width; 
short Height; 
Color bgCol; 
Color figCol; 
bools isOfiiscr; 

}; 

RESULT 

union DnwResp swvitch (VPstat created) { 
caseVPOK: 
Drawable diw; 
default 
void; 

}; 

DESCRIPTION 

Create a new drawable. The drawable could be offscreen or on the screen. The created 
drawable has the foreground and background colors as indicated. 
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IMPLEMENTATION 

With Compound requests the PUSHDRAWABLE can be used. Othenivise with Compound 
requests, subsequent actions would t>e on this drawabie. 

ERRORS 

. iNVAL- If ai^uments exceeds screen parameters- - 
VPNOMOREOFFSCR - if the number of available offscreens exceeds the capability of the 
system 



5.10 FreeDrawable 
SYNOPSIS 

FreeDrawableArgs □ void 

ARGUMENT 

staict FreeDrawableArgs { 
Drawabie drw; 

}; 

RESULT 
Void; 

DESCRIPTION 

This fiees the drawabie and all resources relating to it. 

IMPLEMENTATION 

The associated caches etc. are also freed. 

ERRORS 



5.11 CopyArsa 
SYNOPSIS 

CopyAreaArgs □ CopyAreaRes 

ARGUMENT 

struct CopyAreaArgs { 
Drawabie src; 
Drawabie dst, 
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Short X; 
short Y; 
short Width; 
short Height; 
5 short DstX; 

• short DstY; 

}; 

RESULT 
10 Vpstatstat 



DESCRrPTION 

This allows Copying between two areas either within the same drawables or between 
15 drawables. 

The VP Client copies content from one part of the drawable to another. 
IMPLEMENTATION 

This is a Epical copy request used in Drawing Operations. It could be done either as a simple 
20 ores a compound request 

ERRORS 

INVAL - arguments were invalid 
INVALDRAWABLE - one of the drawables was invalid 
25 ILLEGAL - exceeded the sizes of the drawable 

5.12 ClearRect 
SYNOPSIS 

ClearRectArgs □ VPStat . 

30 

ARGUMENT 

stnjct ClearRectArgs { 

Drawable dnw; 

Short X; 
35 short Y; 

short Width; 

short Height; 

}: 
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RESULT 
Vpstat Stat 

DESCRIPTION 

5 This is used to clear a rectangular area. The area returns to the background color. 



IMPLEMENTATION 

When this area covers the size of the screen, it nnight be prudent to purge the cached 
requests associated with this drawable. 

10 

ERRORS 

- . - -. _ JNVAL - arguments were Invalid 

INVALDRAWABLE - one of the drawables was invalid 
ILLEGAL - exceeded the sizes of the drawable 

15 

5.13 LoadFont 
SYNOPSIS 

LoadFontArgs □ LoadPontRes 

20 ARGUMENT 

stmct LoadFontArgs { 
Drawable drw; 

bools isQuery; r is this a query load V 
Namestr name; /* uses X Font Conventions */ 
25 short Height; 

short Width; 

short type; /* is a combination of FntTypes */ 
}: 

30 RESULT 

struct FontReplystruct { 

short ascent; 

short descent; 

short maxascent; 
35 short maxdescent; 

short width; 

short direction; 

bools isFixed; 

^ ^-Fidfontid; 
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}; 

Union LoadFontRes switch (bool fldreply) { 
case TRUE: 
5 FontReplystaict font; 

casfr FALSE: ■ - . . . .... 

void; 

}; 

10 DESCRIPTION 

This reque^ loads a font for use within a drawabie. The font has a fid which can be used to 
write data. 



IMPLEMENTATION 

15 The Loaded font should confomi to the closest equivalent of the request It might help for 

implementations to keep a table against popular fonts. 

The Fid returned must not be 0. (0 is reserved for the "currenf Fid in compound requests). 

ERRORS 

20 

5.14 FreeFont 
SYNOPSIS 

FreeFontAiigs □ void 

25 ARGUiyHENT 

Stnict FieeFontArgs { 
Drawable drw; 
FidFontid; 

} 

30 

RESULT 
Void; 

DESCRIPTION 
35 Free this Font for this Drawable 

IMPLEMENTATION 

All fonts loaded are freed when a drawable is freed. 
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ERRORS 

5.15 GetTextWidth 
5 SYNOPSIS 

GetTexIWidthAigs-a^tTextWIdthRes- - ..... 

ARGUMENT 

Struct GetTextWidthArgs { 
10 DiBwable Drw; 

Fid fontid; 

shortascent; _ _ _ 

short descent; 

short direction; 
15 short height; 

Uanray ustring; 

}: 

RESULT 

20 Union GetTexIWidthRes switch (Vpstat stat) { 

CaseVPOK: 

Short Length; 

Default 

Void; 

25 } 

DESCRIPTION 

This returns the length of the string on the client's machine. The client allows servers to 
evaluate the right length that is suitable for display. 

30 

IMPLEMENTATION 

VP Clients could implement well known font systems that allow the server to itself decide the 
length of those fonts. ' 

35 ERRORS 

INVAL - aiiguments were invalid 
INVALDrawable - the Drawable is invalid 
INVALFid - there is no knowledge of this Fid 
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5.16 DiawChar 
SYNOPSIS 

DrawCharArgs DVPStat 

ARGUMENT 

struct DiawGharAig8 { - 

Dtawable drw; 
Rd fontid; 
Color col; 
Uchar uchan 
short Ofliset; 
short Lengtii; _ _ 
short X; 
short Y; 

}; 

RESULT 
VPStat; 

20 DESCRIPTION 

Draw a single character at tiie location specified. 

IMPLEMENTATION 

25 ERRORS 

INVAL - arguments were Invalid 
INVALDrawable - the Drawable Is Invalid 
INVALFid - there Is no knowledge of this Fid 

30 5.17 Drawstring 
SYNOPSIS 

DrawStringArgs □ VPStat 

ARGUMENT 

35 . Union Tbstringsvirftch(boolslnTB){ 
Case TRUE: 

Wid TB; 
Case FALSE: 

— '■ Void; '- 
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} 

Struct OrawStringArgs { 
Drawable drw; 
Fidfontid; 
5 Color col; 

Uarray ustring; ------ 

Tbstring Tbstring; 

short Offset; 

short Length; 
10 short X; 

short Y; 

bools isUnderline; 
bools isltalrc; 
bools isSelected; 

15 }; 

RESULT 
Vpstat; 

20 DESCRIPTION 

Write a string at the specified location. The string Is specified in a UTF-16 array. The Tbstring 
Is used to draw Into a TextBox, isUnderLlne if set causes the string to be underlined. Isltalic causes 
It to be Italicized. IsSelected causes it to be selected. 

25 IMPLEMENTATION 

The string must be written in sequence and in the color specified at the location. 
VP Client Implementations must write the data at the location specified. The onus on ensuring 
readability and Layout in the case of VP Clients rests on the server, the software environment and 
the Application. isSelected is implementation dependent 
30 In the case of usage In compound requests, if the Wid is the cunnent Wid (0) then the 

Drawstring is on the last created TextArea, TextFleld or TextBox in the Compound Request. 
Similarty, If the Fid is cunent Fid (0) then the DrawString uses the cunent font that has been loaded. 
ERRORS 

INVAL - arguments were invalid 
35 INVALDrawable - the Drawable is invalid 

IIMVALFid - there is no knowledge of this Fid 

5.18 DrawBytes 

S¥NOPSIS — 
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DiawBytesArgs DVpstat 



ARGUMENT . 

Union Tbstring switch (bools InTB) { 
5 Case TRUE: 

Wld. TB; . . - - - 

Case FALSE: 
Void; 

} 

10 

struct brawBytesAigs { 

Drawabledrw; 

FWfbntid; 

Color col; 
IS Uchararray ustring; 

Tbstring Tbstring; 

short Offeet; 

short Length; 

short X; 
20 short Y; 

bools isUnderline; 
bools isltalic; 

}: 

25 RESULT 

Vpstat; 

DESCRIPTIQN 

DrawBytes is same as DrawString with the difference that the argument is a character array. 

30' 

IMPLEMENTATION 

Same as DrawString. 

ERRORS 

35 

5.19 DrawLine 
SYNOPSIS 

DrawLineArgs □ VPstat 
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ARGUMENT 

Struct DiawUneArgs { 

Dravvable diw; 

Color col; 
5 short X1: 

short Y1; 

short X2: 

short Y2; 

}: 

10 

RESULT 
Vpstat; 

DESCRIPTION 
15 DrawLine draws lines on the specified drawable. 

IMPLEMENTATION 

With compound operations drawline typically will occur on the cun^nt drawable. 

20 ERRORS 

INVAL - aiiguments were invalid 
. INVALDRAWABLE- the drawable was invalid 

5.20. DiawArc 
25 SYNOPSIS 

DrawArcatgs 0 Vpstat 

ARGUMENT 

30 struct DrawArcAiigs { 

Drawable drv^; 

Color col; 

short X; 

short Y; 
35 short Width; 

short Height; 

short StartAngle; 

short ArcAngle; 

J; 
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RESULT 
Vpstat; 

DESCRIPTION 

DiawArc draws an arc at X, Y with the rectangle of width and. height as specified. The 
StartAngie and ArcAngle are In 1/64th of a degree. 
This procedure is also noandatory. 

IMPLEMENTATION 

Some simplistic clients might not have much support for Are Drawing but it is possible to 
simulate It ftom line drawing. Best effort drawing should suffice for Arc Drawing, 

ERRORS 

INVAL - arguments were Invalid 
INVALDRAWABLE - the drawable was invalid 



5.21 FillArc 

SYNOPSIS 

RilArcargs DVPstat 

ARGUMENT 

stnict RilArcargs { 
Drawable dnv; 
Color col; 
short X; 
short Y; 
short Width; 
short Height 
short SteirtAngle; 
short ArcAngle; 

}: 

RESULT 
Vpstat 

DESCRIPTION 
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Similar to DrawArcArgs but provides fill shading. This is again to enable drawing of arcs and 
shapes. 

The procedure is mandatory. 

5 IMPLEMENTATION 

See Above. FIIIDrawing can be simulated by FIIIRects Just as OrBwArc can be simulated wllh 

Lines. 

ERRORS 

10 INVAL- arguments were invalid 

INVALDRAWABLE - the drawable was invalid 



5.22 DrawOvai 
15 SYNOPSIS 

DrawOvalArgs □ Vpstat 

ARGUMENT 

struct DrawOvalReq { 
20 Drawable drw; 

Color col, 
short X; 
short Y; 
short Width; 
25 ^rt Height; 

}; 

RESULT 
Vpstat; 

30 

DESCRIPTION 

Drawing Ovals is also mandatory. X, Y specify the center and the Width and Height specify 
the length and width of the rectangle that fits the Oval. 

35 IMPLEMENTATION 

Could be implemented through multiple arc drawing algorithms. 

ERRORS 

INVAL - arguments were invalid 
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INVALDRAWABLE - the drawable was invalid 

5.23 FlllOval 
SYNOPSIS 

FillOvaiangs □ VpStat 

ARGUMENT 

stnidRIIOvalaigsf 
Drawable drw; 
Color ool; 
short X; 
shortY; 

short Width;' ~ - - 
short Height; 
' >: 

RESULT 
Vpstat 

DESCRIPTION 

Fills the Oval with specified color with Origin X, Y and rectangle of specified width and height 
This procedure is mandatory 

IMPLEMENTATION 

Could be implemented through multiple fill arcs or as a rectangle and several fill arcs. 

ERRORS 

INVAL - arguments were invalid 
INVALDRAWABLE - the drawable was invalid 

5.24 DrawPol^on 
SYNOPSIS 

DiawPolygonargs □ Vpstat 

ARGUMENT, 
struct DrawPolygonReq { 
* Drawable drw; 
Color col; 

short numPoints; : — : : 
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ParrayXpoints; 
ParrayYpoints; 

. ); 

5 RESULT 

Vpslat; 

DESCRIPTION 

The Polygon drawing request specifies Vne number of points and the X and Y coordinates of 
10 the points. A polygon is drawn around them. 

IMPLEMENTATION 

- The lines are drawn In order. The VP Client need not check to see if the coordinates mal« a 
consistent polygon or not. It is upto Servers to specify the order and ensure convexity, etc. 
IS - Polygons is a mandatory procedure. 

ERRORS 

INVAL - arguments were invalid 
INVALDRAWABLE - the drawable was invalid , 

20 

5.25 DrawPolyLlne 
SYNOPSIS 

DrawPolyUneargs DVpstat 

25 ARGUMENT 

struct DiawPolylineReq { 
Drawable dnA^ 
Colored; 
short numPointe; 
30 PanayXpoints; 

PanayYpoints; 

}; 

RESULT 
35 Vpstat 

DESCRIPTION 

The polyline is an open ended polygon (without the last line back to the origin). The pofyline is 
developedfor complex line drawing. - 
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Polyline Is also mandatory as it does not tal<e much beyond line drawing, (altliougli a 
compound request might have sufficed). 

IMPLEMENTATION 

The Polyline Is similar to polygons. 

ERRORS 

INVAL - arguments were invalid 
INVALDRAWABLE - the drawable was invalid 

5.26 FiliPolygon 
SYNOPSIS 

FiliPolygonargs □ Vpstat 

ARGUMENT 

struct FiliPolygonArgs { 
Drawable drw; 
Colored; 

short numPoints; 
ParrayXpoints; 
Parray Ypoints; 

}; 

RESULT 
Vpstat; 

DESCRIPTION 

The fill polygon args request fills a polygon in the specified color. 
This procedure is mandatory. 

IMPLEMENTATION 

Fill Polygon implementations rely on the server to ensure that the points are specified In the 
right order. VP Clients must however be careful about this not being the case as It could cause 
unexpected actions. 

ERRORS 

INVAL - arguments were invalid 
INVALDRAWABLE - the drawable was Invalid 
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5.27 DiBwRect 
SYNOPSIS 

DrawRectaigs DVpstat 

5 ARGUMENT 

struct DiawRectArgs { 
' Dravrable drw; 
Color col; 
short X; 
10 short Y; 

short Width; 
short Height; 
- - ~}\ ' 

15 RESULT 

Vpstat; 

DESCRIPTION 

The DrawRect request is pnsbably one of the most common requests. It is mandatory. 

20 

IMPLEIWENTATION 

It is liltely that DrawRect is a simple and often used routine. With compound request 
implementations, it must be possible to easily perfomn this routine. 

25 

ERRORS 

INVAL - arguments were invalid 
INVALDRAWABLE - the drawable was invalid 

30 5.28 DiawRectTextBox 
SYNOPSIS 

DrawRecfTextBoxArgs □ DrawRectTextBoxRes 

ARGUMENT 
35 Struct for3D{ 

Bools raised; 
Col linecol; 

} 

Union switch 3Drect (bools is3D) { 
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Case TRUE: 

ForSD SDpeFams; 
Case FALSE; . 

Void; 



stnict DrawRectTextBoxArgs { 
Drawable drw; 
Color col; 
10 Color fiHcol; 

Color defaultfbntcol; 
Fid reqFId; 

■ short X; " ~ ' ' 

short Y; 

15 short Width; 

short Height; 

short showfrom; 

3Drect forSDrects; 

t)ools isArea; 
20 }; 

RESULT 

Struct TexffioxReply { 
Wid TBWid; 
25 Fid reqFId; 

short linesdisplayed; 

}; 

union DrawRectTextBoxRes switch (bools created) { 
case TRUE: 

30 TextBoxReply tbreply; 

case FALSE: 
Void; 

}: 

35 DESCRIPTION 

The Rectangular TextBox is the most primitive of Widgets. It is a rectangular area which 
contains text. The TextBox is a rectangular area that displays text that can be scrolled. This Is a 
building block for lists, menus, textfields, text areas, etc. The IsArea parameter indicates if it Is a one 
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line field or an area. The fillcol specifies the color of the box. Showirom Is used to indicate the initial 
place in the text that is cunently shown. 

In Compound operations with OnOp, showfrom can be adjusted on the cached Drawabie. 

The request is not mandatoiy as support for Widgets is not mandatory for compliant VP 
Clients. However, TextBoxes is recommended. It can be used to Ibmn the basis of several other 
widgets and its features can be used to develop many complex devices and systems. 
' The TextBox can be a 30 text box. Refer Draw3Drect 

SetText request is used to set and change the TextArea infonmation. 

IMPLEMENTATION 

This is simple widget which can form the basis for other widgets, it is a building block that VP 
Client implementations could support if they do not have the ability to support the r^st of the 
widgets, This along with compound requests simplifies the making of very good Ul systems. 

With Cached OnOp operations TextBox contents are adjusted using Showfrom. Thus 
Showfrom can be incremented on the Widget 

ERRORS 

INVAL - arguments were invalid 
INVALDRAWABLE - the drawabie was invalid 

5.29 FillRect 
SYNOPSIS 

FillRectArgs □ VPStat 

ARGUMENT . 

struct FillRectArgs { 
Drawabie 6rw, 
Color col; 
short X; 
short Y; 
short Width; 
short Height; 

}; 

RESULT 
Vpstat 

DESCRIPTION - - - - 
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The FiHRect is another commonly used primitive. 
It is mandatory. 

iiy/IPLEMENTATION 
s The RilRect request is a commonly used request 

ERRORS 

INVAL - arguments were invalid 
INVALDRAWABLE - the drawable was invalid 

10 

5.30 DrawRoundRec^ 
SYNOPSIS 

DrawRoundRectargs □ Vpstat 

15 

ARGUMENT 

struct DrawRoundRectargs { 

Drawable drw, 

Color col; 
20 shortX; 

short Y; 

short Width; 

short Height; 

short Ware; 
25 short Hare; 

}: 

RESULT 
Vpstat; 

30 

DESCRIPTION 

DrawRoundRect is anottier commonly used primitive. It consists of several lines and arcs. It is 
a primitive worth supporting due to the fact that well rounded Widgets are very popular for iNJttons, 
etc. It is also mandatory for VP Clients. The Hare and Ware are the Height and Width of the arcs 
35 that Ibnn the RoundRect 

IMPLEMENTATION 
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A round rectangle can be implemented by breaking up the request into a series of lines and 
aros. For certain widget toollcits the use of RR buttons might be very popular depending on the look 
and feel. 

ERRORS 

INVAL- arguments were invalid _ 
- - INVALDRAWABLE - ttie'diBwabie was invalid 

5.31 FillRoundRect 
SYNOPSIS 

FillRoundRectargs DVpstat 



ARGUMENT- - - " ~ ■ - ' 

struct FillRoundRectargs { 

Drawable drw; 

Color col; 

shortX; 

short Y; 

short Width; 

short Height; 

short Ware; 

short Hare; 

}; 

RESULT 
Vpstat; 

DESCRIPTION 

The FillRoundRect fills a roundnect area with the given color. It is a mandatory procedure. 
IMPLEMENTATION 

The Implementation for FillRoundRect typically Involves 4 or 5 FIIIRects and a 4 FIIIArcs. 

ERRORS 

INVAL - argurnents were invalid 
INVALDRAWABLE - the drawable was invalid 

6.32 Draw3Drect 

SYNOPSIS - 
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DrawSDredargs □ Vpstat 

» 

ARGUMEhJT 

struct Draw3DRectReq { 
Drawabledrw; 
Color col; 

shortX ' 
short Y; 
short Width; 
short Height; 
boot raised; 
Color Rgb; 

}; - - - - - • 

RESULT 
Vpstat; 

DESCRIPTION 

Draw3D is to develop a raised or lowered 3D rectangle. It is commonly used in widget 
implementations, textboxes etc. 
This procedure is mandatory. 

IMPLEMENTATION 

VP Clients must fialthfijlly reproduce the server's request. 

ERRORS 

INVAL - arguments were invalid 
INVALDRAWABLE - the drawable was invalid 

5.33 FillSDrBCt 
SYNOPSIS 

Rli3Dreclargs □ VPStat 

ARGUMENT 

struct FillSDRectargs { 
Drawable divr, 
Color col; 
short X; 

— ^ short-Yi 
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Short Width; 
short Height; 
t>ool8 raised; 
Color Rgb; 

}; 

RESULT 
Vpstat; 

DESCRIPTION 

FillSOrect is similar to DrawSD. It colors the interior to reflect a 3D effect on the rectangle. 

IMPLEIVIENTATION- - - - 

VP Clients must faithfully reproduce the server request 

ERRORS 

INVAL - arguments were invalid 
INVALDRAWABLE - the dravvable was invalid 

5.34 Orawlmage 
SYNOPSIS 

Drawlmageargs □ img_reply 

ARGUMENT 

strud Drawlmageargs { 
Drawabledrw; 
imgtype imgfype; 
intTmSize; 
Intsrc)^ 
intsrcY; 
int Width; 
int Height; 
intlmgXid; 
ImgAnay imgarray; 
boots continues; 
bools displayimmediate; 

}; 

RESULT - - - 
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Union Img^reply switch (bools moredetail) { 
Case FALSE: 

img^enums finalreply; 
case TRUE: 

img_enums contreply; 

} ; . , . . . . 

DESCRIPTION 

The Drawimageargs is intended to draw an image and ft perfbrms tlie drawing as indicated. 
There is also the possibility of continuations for Image Requests. Therefore Drawimage has a 
continuation. If the image has a continuation request then the displayimmediate indicates to the 
client whether it must display immediately or not (see implementation on ttie import of ofliscreens) 

Drawimage is a mandatory procedure for VPCIients. 

IMPLEMENTATION 

The implementation requires that Drawimage draw the Image. If the drawable is o^creen, 
then the Implementation could choose to wait till the image is shown to render it to screen at that 

time. 

It is possible to have the first few parts of the image to provide a low level of detail and for 
continuations to provide more detail. 

ERRORS 

INVAL - arguments were invalid 
INVALDRAWABLE - the drawable was Invalid 

5.35 DrawlmageContinuation 
SYNOPSIS 

DrawlmageContinuationargs □ DrawlmageContinuationRes 

ARGUMENT 

struct DrawlmageContargs { 
Drawable dnv; 
imgtype imgtype; 
short srcX; 
short srcY; 
short Width; 
short Height; 
sliorf irngXid^ 
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short Seqnum; 
ImgArray imgarray; 
bools. continues; 
boots displayin^mediate;' 

5 • }: ' 

RESULT 

Union Img^reply switch (bools nnoredetail) { 
Case FALSE: 
10 img^enums finaireply; 

case TRUE: 

Img^enums contreply; 

' ' : ' - 

15 DESCRIPTION 

ImageContinuation offers a continuation to images. Large images are broken up and sent as 
a sequence of packets. There could be a smaller packets or larger packets as appropriate. The 
Sequence number requires that they be sent in order. 

ImageContinuation is mandatory. 

20 

IMPLEMENTATION 

It is possible for continuations to provide added detail. The VP client could indicate that it does 
not want detail beyond a point in an effort to trade off bandwidth and picture quality, given the 
sensifive nature of bandwidth. Some freedom is left to implementations. 

25 . 

ERRORS 

INVAL - arguments were invalid 
INVALDRAWABLE - the drawable was invalid 

30 5.36 DrawLabel 
SYNOPSIS 

DrawLabelArgs □ WidgetU 



ARGUMENT 
35 Struct DrawLabelArgs { 

Drawable dnw; 
Label label; 
Color col; 
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RESULT 



WidgetU; . 
5 DESCRIPTION 

This Is used to define Labels. These.are important- Widget operations. Labels are passive 
controls and do not support user interaction. As Widget support is optional, support for this 
operation is optional. However VP Clients that support Widgets must support Labels. 

10 IliflPLEIWIENTATION . 

The implementation for Labels oould vary between VP Clients. However, they must strive to 
implement support for the behavior specification expected. . 



ERRORS 



15 



INVAL - arguments were invalid 
IIWALDRAWABLE - the drawable was invalid 



20 



5.37 DrawButton 
SYNOPSIS 

OrawButtonargs □ WidgetU 



ARGUMENT 



Union Widsize switch (bools recommend) { 
Case TRUE: 



25 



RectSize 



size; 



Case FALSE: 



30 



Void; 

}: 

Union Location switch (bools recommend) { 
Case TRUE: 

Coordinate Place; 
Case FALSE: 
Void; 

}; 



35 



stnrct Button { 
Label label; 

}: 



struct OrawButtonargs { 



Drawable drw; 
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Button button; 

Location where; 
Widsize Wldgetsize; 
Color col; 
5 }; 

RESULT 
WidgetU 

10 DESCRIPTION 

The DrawButton request draws a button to a specified drawable. The Label associated with a 
. . . .button can be changed using SetTe)d later. Thus an empty-La^ 

possible. Location and Size could spedfy the location and size of the 
Button. 

15 ' DrawButton Is not mandatory. 
IMPLEMENTATION 

The implementation of Widgets could be very system specific. For instance, buttons can be 
square, rectangular, oval or roundrects. VP Clients are at a liberty to implement these in any way. 
20 The location, size recommendation is however important as VP Clients must strive to offer servers 
the ability to handle layout Widget Implementations must also restrict the Label to be within the 
Widget, so partially displayed Labels are allowed. 

ERRORS 

25 INVAL - arguments were invalid 

INVALDRAWABLE - the drawable was invalid 
VPNOTSUPP - Widget support is absent 

5.38 DrawCheckbox 
30 SYNOPSIS 

DrawCheckboxargs □ WidgetU 

ARGUMENT 

union CbWidU switeh (bool partofGroup) { 
35 case TRUE: 

intWidU; 
case FALSE: . 
void; 
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5 



Struct Checkbox { 
Label label; 
boo! on; 

CbWidU checkboxgroup; 

}; 

struct DrawCheckboxargs { 
Drawable drw; 
Checkbox checkbox; 
Widsize size; 



10 



Location where; 



Color col; 

}; 



RESULT 



15 



WidgetU 



DESCRIPTION 

The Checkbox request is intended to create and draw a check box. "On" sets the checkbox 
on. The Checkbox could be part of a Checkbox group in which case it is added to the 
20 Checkboxgroup. If the Checkbox is set and the Checkbox is part of a Checkbox group, then the new 
Checkbox is set and the already on Checkbox disabled. 

IMPLEMENTATION 

25 The Implementation for Checkbox could be specific to the VP Client implementation. 

However the size recommendation is important 



ERRORS 



30 



INVAL - argumente were invalid 
INVALDRAWABLE - the drawable was invalid 
VPNOTSUPP - Widget support is absent 



35 



5.39 DrawCheckboxGroup 
SYNOPSIS 

CheckboxGroupargs □ WidgetListU 



ARGUMENT 



staict CheckboxGroupargs { 



Drawable dnv; 



wo 01/91482 



PCT/USOl/17274 



Checkbox cblist<>; 
Color col; 
}: 

RESULT 
5 WidgetUstU; 



DESCRIPTION 

The Checkboxgioup request is intended to create a nnutually exclusive set of checkboxes. 
Only one of the list can be set to on. The list can be empty as well and individual Checkboxes can 
10 be added later. 

IMPLEMENTATION 

The VP Client implementation must check to ensure that only one of the Checkboxgroup is 
checked on. The VP Client must strive to use the recommendattons for size and location as this 
IS enables layout handling at the Server. 



ERRORS 



20 



ilWAL - arguments were invalid 
INVALDRAWABLE - the drawable was invalid 
VPNOTSUPP - Widget support is absent 



5.40 DrawChoiceList 
SYNOPSIS 

DrawChoiceListargs □ WidgetU 



25 



ARGUMENT 



30 



staictChotoeltem{ 
Uarray Name; 
}: 

staict DrawChoiceListaigs { 



Drawable dro^ 



35 



intselectedindex; 
ChoiceHem Listo; 
Location where; 
Widsize Widgetsize; 
Cotor col; 

}: 



RESULT- 
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DESCRIPTION 

The Implementatfon for ChoioeList is to create a list of items tliat can pop up and give the user 
5 the ability to choose from that list There is a default selection. (1 ... .size 
. . . Of list). The value -1 indicates that-there is no selection. Future references could use 1 . .. size Of list 
to select an item. 

IMPLEMENTATION 

10 VP clients could specify an appropriate Implementation for this Widget VP clients could also 

choose the default when the -1 index is set (to importune the user to select). 



ERRORS 

INVAL - arguments virere invalid 
15 INVALDRAWABLE^ the drawable was invalid 

yPNOTSUPP - Widget support Is absent 

5.41 Dravi^st 
SYNOPSIS 
20 DrawListargs □ WidgetU 

ARGUMENT 

struct Listltem { 
Uarray Name; 
25 bools selected; 

}; 

struct DrawListargs { 
Drawable 6rw, 
, bools multiSelec^ 
30 Listltem Listo; 

Location \where; 
Widsize WIdgetsize; 
Colored; 

}; 



35 



RESULT 

WidgetU; 



DESCRIPTION 
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DrawList request is similar to Choice, except that multiple selections are possible. Several 
selections are possible. 

IMPLEMENTATION 

5 The implementation typically requires a scrollbar and the ability to scroll down the list and 

niake a selection. Several Items are selectable. - - - 

ERRORS 

INVAL - arguments were invalid 
10 INVALDRAWABLE - the drawabie was.invalid 

VPNOTSUPP - Widget support is absent 

— -5:42 DrawScrollBar 

SYNOPSIS 
15 DrawScrollBarsargs □ WidgetU 

ARGUMENT 

enum ScrollbarType { 
VERTICAL = 0, 
20 HORIZONTAL .= 1 

}: 

enum ScrollType { 
PAGEUPDOWN, 
LINEUPDOWN, 

25 }; 

staict Scrollbar { 
ScrollbarType stype; 
ScrollType atype; 
short InitialValue; 
30 short ThumbSize; 

short Min; 
short Max; 
}: 

sbtjct DrawScrollBaraigs { 
35 Drawabte dnv; 

Scrollbar sb; 

Location where; 
Widsize Widgetsize; 
Bools DeliveiScrolllnfoy 
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Color col; 

}: 

RESULT 
WidgetU 

DESCRIPflON 

DrawScroliBar chaws a scrollbar that <^n scroll up or down. It Is t/plcaliy coupled with other 
Widgets. ScrollBars form a sliding scale of adjustment The ScrollType decides whether a single 
touch leads to a large block move or a single line move. 

Deliver Scrolllnfb forwards scrolling events back to the server. 

IMPLEMENTATION 

Scroll Bars can work with Compound Operattons and OnOp combinations requests to develop 
common Widgets Implementattons: They implement the ONScrollLUP, ONScrollLdown. 
ONScrollPUP, ONScroilPDown operations. 
Using Compound Requests one could, 
OrawRectTextBox 
PushWid ' . 

DrawScrollBar 
OnAction (currentWid) - 

ONScrollPUP - PopWid - incr-ement Showfrom 
ONScroilPDown - PopWid - Decrement Showfrom 
OnAction (PopWid) - 

OnSelect - SelecAText (and HighLight) SendEvent 

ERRORS 

INVAL - arguments were invalid 
INVALDRAWABLE - ttie drawable was invalid 
VPNOTSUPP - Widget support is absent 

5.43 DrawTextFleld 
SYNOPSIS 

DiawTexFieldargs DWkigetU 

ARGUMENT 

struct indexrange { 

int start; 

int^nd^ ~ ~ 
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}; 

union selectedTextU switch (bool selected) { 
case TRUE: 

indexrange irange; 
5 case FALSE: 

void; . - - ■ ' - 

};" 

struct DrawTextFieldargs { 

Drawable dm; 
10 bool isEdiiable; 

bool seEchoChar, 

intnumChars; 
Uanray stn 

SelectedTextU stU; 
15 Color col; 

}; 

RESULT 

WidgetU; 

20 

DESCRIPTION 

The DrawTextField request is intended to support the creation of TextFields. These are single 
line text areas and can be used for Inputing text or to display a line of Text 
This is an optional procedure. 

25 

IMPLEMENTATION 

The TexlField is implemented to support text entry. They are straightfonward implementations 
of simple text handling. Several contemporary User Interface Systems have support for TextFields 
and Areas. 

30 

ERRORS 

INVAL - arguments y^ere invalid 
INVALDRAWABLE - the drawable was invalid 
VPNOTSUPP - Widget support is absent 

35 

5.44 DrBwTextArea 
SYNOPSIS 

DrawTexlAreaargs □ WidgetU 
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ARGUMENT 

enum TareaScrollbars { 

SCROLLBARS^BOTH, 

SCROLLBARS^NONE, 
5 SCROLLBARS^HORIZONTAL.ONLY. 

SCROLLBARS J/ERTICAL^ONLY - 

stnicf DrawTextAreaargs { 

Drawabie dnA^ 
10 TareaScrollbars tsb; 

short numUnes; 

short numChars; 

short showfrom; 

bools isEditabie; 
15 bools DeliverScroil; 

Uarray str; 

SelectedTextU stU; 

Color col; 

}: 

20 

RESULT 

WIdgetU 

DESCRIPTION 

25 The TextArea request is for creating TextAreas. Typically they are square areas where text 

scrolls. The associated scrollbar can be used for scrolling through the text 
The request specifies the number of lines and characters and a portion of the text can be selected. 
This is an optional procedure. 

30 IMPLEMENTATION 

TextAreas are commonly used primitives. It is typical in mos;t applications, notably browsers, 

etc. 

ERRORS 

35 INVAL - argumente were invalid 

INVALDRAWABLE - the drawabie was invalid 
VPNOTSUPP - Widget support is absent 

5>15~DrawR/leriu "~ 
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SYNOPSIS 

DrawMenuArgs □ WidgetListU 

ARGUMENT 
5 enum MenultemType { 

MENUITEM = a. . - - - - 

SUBMENU = 1, 
CHECKBOXMENUITEM = 2 

}; 

10 struct Menul { 

Label label; 
bool isEnabled; 

}: " 

struct CheckboxMenul { 
IS Label label; 

bool pn; 

}: 

union Menultem switch (MenultemType type) { 
case MENUITEM: 
20 Menul menui; 

case SUBMENU: 

Menu menuelem; 
case CHECKBOXMENUITEM: 

CheckboxMenul cbmenui; 

25 }; 

struct Menu { 

Label name; 

bool removable; 

Menultem menulisto; 
30 }; 

stmct DrawMenuargs { 

Drawable dryr, 

Menu menu; 

Color col; 
35 }; 

RESULT 

WidgetListU; 
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DESCRIPTION 

DrawMenu is intended to support Menus. A menu contains a list of available items and the 
items could have other submenus under them. The Menu Is a complicated and exploiabie Selection 
list and is useful in client devices with small screens (but not too small). 

The reply returns a list of Widgets. The Widgets conespond to the list of Widgets created and 
in the same order as that requested. 

DrawMenu is an optional request 

IMPLEMENTATION 

The implementation can be quite complicated In some systems as there is limited real estete 
on most handsels. However when available It is a useful feature. Applications must limit the size of 
menus to small and measureable sizes. 

The implementation must also support the conresponding event selection and handllr^. 

ERRORS 

INVAL - argumente were invalid 
INVALDRAWABLE - the drawable was invalid 
VPNOTSUPP - Widget support is absent 

5.46 DrawMenuBar 
SYNOPSIS 

DrawMenuBarargs □ WidgetListU 

ARGUMENT 

stmctMenuBar{ 
Menu menuso; 
}: 

struct DrawMenuBarArgs { 

Drawable drw; 
MenuBar mb; 
Color col; 

}: 

RESULT 

WidgetUstU; 

DESCRIPTION 

MenuBar are horizontel bars that contain a list of Menus. These are typically for larger screen 
systeTfiOTMiiiuBarisa series of Menus. 
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MenuBar is completely optional. 
IMPLEMENTATION 

The Implementation for MenuBar can be quite complicated. There are several contemporary 
5 Ul systems that support MenuBars. Thus MenuBar is supported in VP as well. MenuBar 

implementation can be avoided to simplify software burdens on most handset implementations. 



ERRORS 



INVAL - arguments were invalid 
INVALDRAWABLE - the drawable m^s invalid 
VPNOTSUPP - Widget support is absent 



5.47 ' SetText 

SYNOPSIS 



15 



SetTextArgs □ Vpstat 



ARGUMENT 



union SetTextU switch (SetTextHow how) { 



20 



case START: 

Uan^y str; 



case APPEND: 

Uanray str; 



case INSERT: 

Insertstr istr; 



25 



case REPLAGERANGE: 
Replacestr rstr; 



}: 

struct SetTextargs { 
Drawable drw; 



30 



short Wid; 

Color col; 



SetTextU stu; 
Bools highlight; 
Bools Italicize; 
Bools Underline; 

}: 



RESULT 

Vpsfel; 
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DESCRIPTION 

The SetText request adds text to a specified Widget. It is possible to do SetText on a Textbox, 
TextReld,.eto. 

5- 

IMPLEMENTATION 

ERRORS 

INVAL - arguments were invalid 
10 INVALDRAWABLE- the dtawable was invalid 

VPNOTSUPP - Widget support Is absent 
INVALWID - The Wid is invalid 

INVALTEXTOISIWID -Text Additions and actions spedfled are not possible on this Wid 

15 5.48 ClearText 
SYNOPSIS 

ClearTextArgs OVpstat 



ARGUIVIENT 
20 Struct Textrange{ 

Short index; 
Short length; 

} 



25 enumsClearTextHow{ 
ALL=0, 
FR0M=1 

}: 

Unfon ClearTextU switch (ClearTextHow cth) { 
30 Case ALL: 

Void; 
Case FROM: 

Textiangetn 

} 

35 struct ClearTextergs.{ 

Drawable dtw; 
short Wid; 

ClearTextU ctu; 
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RESULT 
Vpstat; 

DESCRIPTION 

Clearjext cleaifs the text In an a. VWdget 

IMPLEMENTTATION 

SetText can itself be used for much of the functionality of ClearText. 

ERRORS 

INVAL - arguments were invalid 
INVALDRAWABLE Vthedrawable was Invalid 
VPNOTSUPP - Widget support is absent 
INVALVyiD - The Wid is invalid 

INVALTEXTONWID - Text Additions and actions spedfled are not possible on this Wid 

5.49 SelectText 
SYNOPSIS 

SelectTextArgs □ Vpstat 

. ARGUMENT 
Enums PositionHow { 
CHARGOUNT = 0, 
UNEC0UNT=1. 
LINEANDCHAR=2 

}; 

stnjct lineandchar { 
short llneno; 
short chamo; 

}: 

Union Position switch (enum PositionHow) { 
Case CHARCOUNT: 

Short chamo; 
Case LINECOUNT: 

Short lineno; 
Case LINEANDCHAR: 

Lineandchar Inc; 
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Struct SelectTextArgs { 
Drawable drw; 

short Wid; 
Color col; 
5 PositionHow ph; 

Short Length; . 

} 

RESULT 
10 Vpstat; 

DESCRIPTION 

This Is used to select a part of the Text A selection can occur or be set on any part of the text 
This is optional but if a VP client implementation Implements any of the Text Widgets 
IS specified it must implement SelectTexL 

IMPLEMENTATION 

The selected text is highlighted. The highlighting is position Independent 

ERRORS 

INVAL - arguments were invalid 
INVALDRAWABLE - the drawable was invalid 
VPNOTSUPP - Widget support is absent 
INVALWID - The Wid is invalid 

INVALTEXTONWID - Text Additions and actions specified are not possible on this Wid 

5.50 SelectonWidget 
SYNOPSIS 

SelectOnWidgetargs □ Vpstat 

30 

ARGUMENT 

Enums Select { 
THISWID = 0. 
INDEXNUMsl, 
35 LABELSTRING = 2 

}; 

Union SelectHow switch (select si) { 
CaseTHISWiD: 
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Case INDEXNUM: 

Short indmc; 
Case LABELSTRING: 

Sting label; 

5 } 

SelectOnVVidgeta^s{ 

Orawable drw; 

WId wid; 
SelectHowsh; 
10 }; 

RESULT 

- Vpstat; " ' 
15 DESCRIPTION 

The SelectOnWIdget request causes a selection to be made on a Widget Item or the Widget 
itself. This is used in Widget implementations of various kinds. 

IMPLEMENTATION 
20 The implementation requires a highlighting of the item chosen, 

ERRORS 

INVAL - arguments were invalid 
INVALDRAWABLE - the drawable was invalid 
25 VPNOTSUPP - Widget support is absent 

INVALWID - The Wid is Invalid 

INVALACTIONWID - Actions specified are not possible on this Wid 

5.51 AddtoWidget 
30 SYNOPSIS 

AddtoWidgetargs □ Vpstat 

ARGUMENT 

Enums itemType { 
35 CHECKBOX sO, 

CHOICEITEM = 1, 
LISTITEM=2, 
MENUITEM = 3, 

MENU = 4- ■ 
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>: 

union HemWhich case (Itemtype it) { 
casd CHECKBOX: 
5 Chedcboxcb; 

CaseCHOICEITEIW: 

Choiceltem ci; 
Case LiSTiTEI\/l: 
Listitem li; 

10 CaseMENUITEI\A: 
Menultem mi; 
Case MEm: 

~ - - - MENOme:" ' " " 

}; 

15 

AddtoWidgetafigs { 
Drawabledrw; 

sliortWid; 
itemWhjch iw; 

20 ); 

RESULT 
Vpstat- 

25 DESCRIPTION 

The AddtoWidget prooedure is intended to add a new item or widget to a given widget It can 
be used fw adding items in response to actions etc. 

This is optional as it applies to VP Clients that supports Widgets. 

30 IMPLEMENTATION . 

Selections and Additions are present to enable VP Clients to add to items to Widgets. 

ERRORS 

INVAL - arguments were invalid 
35 INVALDRAWABLE - the drawable was invalid 

VPNOTSUPP - Widget support is absent 
INVALWID - The Wid is invalid 

INVALACTIONWID - Actions specified are not possible on this Wid 
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5.52 RemoveFromWidget 
SYNOPSIS 

RemoveFromWidgetArgs □ Vpstat 

5 

ARGUMENT - - - . - . . . 

union RemoveltemWhich case (Itemtype it) { 
caseCHECKBOX- 

Wid Widget; 
10 Case CHOICEITEiy/l: 

Short index; 

- - Case USriTEM:. - - - _ . 

Short index; 
CaseMENUiTEIM: 
IS Wid Widget; 

Case MEm: 

Wid Widget; 

}; 

20 RemoveFromWidgetaigs { 

Drawable dnv; 

short Wid; 
RemoveltemWIiich iw; 

}; 

25 

RESULT 
Vpstet; 

DESCRIPTION 

30 RemoveftomWidget renroves an item from a Widget wid. 

liy/IPLEiyAEI^ATiON 
ERRORS 

35 INVAL- arguments were Invalid 

ilMVALDRAWABLE - the drawable was invalid 
VPNOTSUPP - Widget support is absent 
INVALWID - The Wid is invalid 
— -INVAtACTIONWID" - AdSonsspedfied are^ M 
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5.53 NewApplication 
SYNOPSIS 

NewApplicationargs □ VPstat 

5 

ARGUMENT 

union ClassmarkU switch (boois isMExE) { 
case TRUE: 

short classmarl<; 
10 case FALSE: 

void; 

}: 

Stnict Program { 
15 ClassmarkU CU; 

short Environment; 
Featureset featureso; . 
LocationURL. programURL; 

}: 

20 

Struct Server { 
Address Vphost; 

} 

25 struct NewApplicationargs { 

Program program; 
Server recommendedserverso; 
Bools exiting; 

} 

30 

RESULT 
Vpstat; 

DESCRIPTION 

35 The NewApplication request takes place sometimes on a click or some other event The VP 

Server, realizes that this leads to a new program launch, Infonms the VP Client that it has requested 
the Launch of a new application. The VP Client then uses the ALM Protocol to launch a new 
application. 

TtteWiaBleexiting"indicates that the current application Is exiting. 
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IMPLEMENTATION 

The VP Client must check the list of servers. If the list is empty then the VP Client uses 
NewAppilcation to contact the ALM to launch the Application on any appropriate server and waits for 
5 the VP Server to connect to it. If the list Is not empty, the the VP Client uses LaunchNewOnServer to 
get the ALM to launch the application on a given server. The Application (or Qie^nvlronment in 
which It runs) will then launch Itself arid use this protocol to connect to this VP client 

The VP Client can also reply saying that there Is no launch. In this case the cunent application 
will continue to drive the Ul or do a APPLICATIONCHANGE back to the ALM. 
10 If the cunent application is exiting, the VP sen/er informs the ALM before It exits. 

ERRORS 

~ ' VPiNVAL - aiiguments were Invalid 

VPNOTLAUNCH - the VP Client does not want to Launch a new appiteatlon 

15 

5.54 CopyDrawable 
SYNOPSIS 

CopyDrawableargs □ CopyDrawableRes 

20 ARGUMENT 

Struct RDrawable { 
Drawabie drw; 
Boois replace; 

}; 

25 

Union DestDnw switeh (boois create) { 
CaseTRUE: 
Void: 
Case FALSE: 
30 RDrawable DestDrw; 

} 

Struct CopyDrawableargs { 

Drawabie srcdnv; 
35 DestDrw dstelnw; 

Boois IsOfiscr; 
} . 

RESULT - 
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Union CopyDrawableRes switch (VPStat vp) { 
caseVPOK: 

Drawabie dstdrw; 

Delault 

Void; 

DESCRIPTION 

CopyDrawable copies the contents of a drawabie. The srcdrw can be the rootdrawable or an 
offscreen drawabie. 

the destination drawabie must also be held for subsequent requests and is active. VP clients 
must copy to specified destination drawabie unless the replace Is TRUE. If Replace is TRUE, then 
the VP Client can choose to either replace the drawabie or not The request returns the old or new 
drawabie as appropriate. 

IsOffecr specifies if the destination is onscreen or offscreen. 

This operation is optional but it is a must for implementations with Compound Operations. 
IMPLEMENTATION 

The copy action can be a cache request copy. Copy Drawabie could be used along witti 
Compound Requests to alter a given viewable screen and show it witii less use of bandwidtti. 
Thus the following Compound Operation, 
PushDrawable (cun-ent root) 

CopyDrawable (root to created) -^(created drawabie will become cun-ent) 
Drawing and Screen Actions 

ShowDrawable (showtiie cunrent which becomes new root) 

PopDrawabie (old root) 

FreeDrawable (free ttie old root drawabie) 

ERRORS 

INVAL - arguments were invalid 
INVALDRAWABLE - flie drawabie was invalid 
VPNOTSUPP - Widget support is absent 



5.55 PushDrawable 
SYNOPSIS 

PushDrawableargs DVpstat 

ARGUMENT- 
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Struct PushDtawableargs { 
Diawable drw; 

} 

5 RESULT 

Vpstet; . . _ . . . . . - - . - - - . - - - 

DESCRIPTION 

The request has relevance only In the case of Compound Operations. It allows a push of the 
10 Drawable onto the stack or save area of drawables. 

Optional for those VP clients that do not support Compound Operations. 

IMPLEMEWTAtlON 

The implementation for this requires that drawables be pushed and popped as needed. 

15 

ERRORS 

IIWAL - argument were invalid 
INVALDRAWABLE - the drawable was invalid 
VPNOTSUPP - No Support for ttiis procedure in this VP Client 

20 , 

5.56 PopDrawable 
SYNOPSIS 

Void □ PopDrawableres 

25 

ARGUMENT 
Void; 

RESULT 

30 Union PopDrawableRes switch (Vpstat){ 

CaseVPOK: 

Drawable dnv; 

Default 

Void; 

35 } 

DESCRIPTION 

Tlie request has relevance only in the case of Compound Operations. It allows a pop of the 
Drawable onto tiie8tacl<orsa\^ area of clr^^ ' 
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Optional for those VP clients that do not support Compound Operations. 
IMPLEMENTATION 

The Implementation for this requires that drawables be pushed and popped as needed. 

5 

ERRORS 

INVAL - arguments were Invalid 
INVALORAWABLE - the drawabie was Invalid 
VPNOTSUPP - No Support fbr this procedure in this VP Client 

10 

5.57 RepaintDrawable 
SYNOPSIS 

RepaintDrawaibleArgs 0 Vpstat 

15 ARGUMENT 

Union SubRect switch ( bools isSubRect) { 
Case TRUE: 

Rectangle subrect; 
Case FALSE: 
20 Void; 
} 

Struct RepaintDrawableArgs { 
Drawabie drw; 
25 SubRect sb; 

J: 

RESULT 
30 Vpstat; 
DESCRIPTION 

The RepaintDrawable request is used to repaint the drawabie or a portion of it 
This is again Optional. 

35 

IMPLEMENTATION 

The RepaintDrawable request is used to refresh a drawabie and could happen as part of a 
compound request The Compound Request could lead to the Drawabie being updated with 
requests and then the entire-fc^^fTraquests could'be replayed or refieshed. 
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Repainting portions of screens may not be possible In some Ul systems. 

ERRORS 

INVAL - arguments were invalid 
5 INVALDRAWABLE - the drawabie was Invalid 

VPNOTSUPP - No Support for this procedure In this VP Client - - 
VPCACHEREPLAY - The Cache of request was lost; replay the dravi^ble cache 

5.58 ShowDrawabie 
10 SYNOPSIS 

ShowDrawableArgs DVpstat 



- ARGUMENT - - - " 

.Stmct ShowDrawableArgs { 
15 Drawabie 6iw, 

} 

RESULT 
Vpstat; 

20 

DESCRIPTION 

The ShowDrawabie request Is used to show a drawabie to the screen. This causes an 
offiscreen drawabie to be made the root drawabie. 

25 This procedure is mandatory. 

IMPLEMENTATION 

The ShowDrawabie request can sometimes cause a VPCACHEREPLAY error. The VP Client 
Issues this error when it has discarded the cache for the drawabie. The Sender will then attempt to . 
30 recreate the state of the drawabie on the client 

ShowDrawabie could be the last procedure In a compound request to update the screen with 
the oontente of a newly created drawabie. 

ERRORS 

35 INVAL -ai^umente were invalid 

INVALDRAWABLE - the drawabie was invalid 
VPNOTSUPP - No Support for this procedure in this VP Client 
VPCACHEREPLAY - The Cache of request was lost; replay the dravi«ble cache 
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5.59 MapSubDrawable 
SYNOPSIS 

MapSubDrewableArgs DVpstat 

5 ARGUMENT 

Stmct MapSubDiBwableArgs { ... - ■- 

- Drawable diw; 
Coordinate Location; 

} 

10 . 

RESULT 
Vpstat; 

DESCRIPTION 

15 The IVIapSubDrawable is for VP client implementations that have a Windowing system or 

minlwindowing system. They map the subdrawable to the specified location which specifies the top- 
left comer. 

This is purely optional. . 

20 IMPLEIWENTATION 

This is for VP Clients that have Windowing systems implemented. 

ERRORS 

INVAL - arguments were invalid 
25 INVALDRAWABLE - the drawable was invalid 

VPNOTSUPP - No Support for this procedure in this VP Client 

5.60 UnMapSubOrawable 
SYNOPSIS 

30 UnM^pSubDrawableArgs □ Vpstat 

ARGUMENT 

Struct UnMapSubDravwableArgs{ 
Drawable drw; 

35 }; 

RESULT 
Vpstat 



-102- 



wo 01/91482 



PCT/USOl/17274 



DESCRIPTION 

The UnMapSubDrawable is a request Intended to unmap a given subdrawable. It perfonns 
this operation and malces the drawable an offscreen drawable. 

Purely optional, but all VP clients that implement MapSubDrawable must implement 
5 UnMapSubDrawable. 

IMPLEMENTATION 

The UnMapSubDrawable is for VP Clients that have Windowing Systems or similar 
fiinctibnality. There is no requirement to cache the Drawable, but it is better to do so as It might be 
10 brought bade to be shown. 

ERRORS 

- - INVAt -arguments were Trivalld 

INVALDRAWABLE - the drawable was invalid 
15 VPNOTSUPP - No Support for this procedure In this VP Client 

5.61 ListCachedDrawables 
SYNOPSIS 

ListCachedDrawablesargs □ ListCachedOrawableRes 

20 

ARGUMENT 
Void; 

RESULT 

25 Union ListCachedDrawableRes switch (Vpstat stat) { 

CaseVPOK: 

Drawables dnWisto; 

Default 

Void; 

30 }; 

DESCRIPTION 

The LislCachMlDrawabies request is to help in the management of the drawable caches. The 
VP Client replies with the list of drawables that It holds in Its cache (in ftili) for this application 
35 session. 

IMPLEMENTATION 

The ListCachedDrawables is used by VP Servers to detemiine which drawables to replay to 
ameliorate end user performance-delays: This is typiialalTOa^ 
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ERRORS 

• INVAL- arguments were invalid 

VPNOTSUPP - tliis procedure is not supported 



5.62 PushWid 
SYNOPSIS 

Wid avoid 

ARGUH/IENT 
Wid 

RESULT 
Void 

DESCRIPTION 

PushWid pushes a Widget on the stack of Wids. It assumes Compound Operations. 
This is an optional procedure. 

IMPLEMENTATION 

This is used to enable Compound Operations. 

ERRORS 

INVAL - arguments were invalid <• 
VPNOTSUPP - this procedure is not supported 

5.63 PopWid 
SYNOPSIS 

Void oWidU 
ARGUIWIENT 
Void; 

RESULT 

Union WidU switch (VPStat stat) { 
Case FALSE 
Void; 
Case TRUE: 

^Nid — wid.' 
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}: 

DESCRIPTION 

The PopWId primitive pops the WId and makes It the current WId. 

5 

IMPLEMENTATION ...... 

The use of PopWId is to change the WId in Compound Operations. 

ERRORS 

10 VPWiDSTACKEMPTY- the WIO stack was empty 

5.64 PushFid 
■ SYNOPSIS 

Fid DVoid 

15 

ARGUMENT 
Rd 

RESULT 
20 Void; 

DESCRIPTION 

This pushes a Fid onto the stack. The PushFid request is used in Compound Operations. 
This is an optional request 

25 

IMPLEMENTATION 

The use of PushFid/PopFid is to enable Compound Request to VP Clients MultlFont 
capabilities to efficiently add text that is in several fonts. 

Notably It allows Textareas like those in a browser to send single requests to update a full 

30 page. 

ERRORS 

VPINVALFID - The Fid is invalid 

35 5.65 PopRd . 
SYNOPSIS 
Void 0 Fid 

ARGUMENT ' * " " " " 
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Void: 

RESULT 
Fid; 

5 ... 

. . DESCRIPTION ----- 

PopFid pops the Fid from the stacit 

IIMPLEMENTATION 
10 " See atjove. 

ERRORS _ _ . . 

VPFIDSTACKEMPTY - The FID stacit was empty (underflow) 

15 5.66 ClearScreen 
SYNOPSIS 

ClearScreenargs □ VPStat 

ARGUMENT 
20 Struct ClearScreenargs { 

Bools ReieaseMappedDrawables; 
Bools ReleaseAIIDrawables 

} 

25 RESULT 

VPStat; 

DESCRIPTION 

The ClearScreen request is intended to clear the screen. If ReieaseMappedDrawables is set, 
30 then all subdrawables that are mapped cun-ently are released. If ReleaseAIIDrawables is set then 
all Drawables that are both Mapped and Unmapped are released. 

IMPLEMENTATION 

Clear Screen also causes the freeing of Drawable Caches associated with the root drawable 
35 as well as mapped subdrawables. 

ERRORS 
""srST^AppliratlonChange 
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SYNOPSIS 

ApplrcationChangeargs □ Vpstat 

ARGUMENT 

5 Union AppServerU switch (bods NewApp){ 

Case TRUE; _ 

Program program; 
Case FALSE: r an existing application */ 

Appiicationlnstance ai; 

10 } 

Struct AppiicatlonChangeargs { 
Chiang^Sen^arU appf 

Address server, 
15 Bools exiting; 

Bods isAUy^; 

}; 

RESULT 
20 Vpstat; 

DESCRIPTION 

The ApplicationCiiange request directs tine VP Client to cliange servers. It requests the VP 
Client to switch to a new application possibly on a different server (by interacting with the ALiyfl). This 
25 could lead to the launch of a new appiication on the server or reconnection with an existing 
application on the server based on what was request 

The ApplicationChange request could also be back to the ALM. if isALM is set 
If exiting lis set, it Implies that the current application is exiting. 

30 IMPLEMENTATION ^ 
Similar to NewApplication but this could also be to an e)dsting appiication. 
The ApplicationChange request could also be bacic to ttie ALM. The ALM can then either 
If exiting is s^ all the caches assodated with Oils Server must be destroyed. 

35 ERRORS 

VPINVAL - the arguments were invalid 

VPCHANGEFAILEO - the launch or change to the server specified failed 
5.68 - OnAction Operation- ^ - - 
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15 



20 



25 



30 



SYNOPSIS 

OnAcBonArgs 0 VPStat 

ARGUMENT 

union VP_arg8 switch (procnums proc) { 
. . Case <OPCODE>: <argument>; 



struct Compoundatgs { 
VP_args VP_argarray<>; 

} 

StarJet SlmpieArgs{ 
VP_arg8VP_arg: 

}: 

union Actionargs switeh (txiois compound) { 
case TRUE: 

Compoundargs cp; 
Case FALSE: 

SimpleArgs sp; 

} 

enums OnOpType { 
ONCLICK=0, 
0N0VER=1, 
0N0UT«2. 



0NSELECT«4, 
OnScroliLUP»5. 
OnScroliLDOWN a 6, 
OnScroliPUPs?, 
OnScroilPDOWN « 8 

}; 

union AcBonProc switch (OnOptype operation) { 
case <OPTYPE>: 




Axstionargs Action; 
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> 

Union ObjU switch (bocils isWid) { 
Case TRUE: 

Wld .widget; - - 

Case FALSE: 
Void; 

}' 

StnictWidAction{ 
ObjU whiciiObj; 
ActionProc actonWid; 

} 

OnActionArgs { 
WidAcfion widlisto; 

> 

RESULT 
VPStat; 

DESCRIPTION 

The OnAction request is a persistent request that is used along with Compound Operations. It 
acts as a script for actions that tal<e place on the VP Client 

The VP Client caches the OnAction request and returns after evaluating if the request can be 
reasonably handled on the VP Client 

There are two types of OnAction requests. Those that tal<e place on Widgets and those that 
take place on primitive requests, (like reds, lines, text etc.) In the case of the latter, they could lead 
to highlighting underlining, etc. 

OnAction requests cannot nest 

OnAction woria with ChangeVarlable to also acoortpllsh additional actions. 
This is a purely optional capability. 
Editor's Note: Alternate mechanisms are possible for Implementing OnAction style capabilities. 

IMPLEMENTATION 

OnAction is a more intricate capability that requires interaction with the Event System etc. It is 
present mainly to better support Client side scripting, (as In browsers etc.). 
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The OnAction Implementation must be careful to prevent deadlocking. Tiie typical 
Implementation will create a set of checks that the Event System peruses on User Actions. If a User 
Actions is on a Widget that requires an OnAction action, appropriate action is taken. In the case of 
Widgets more complicated capabilities are possible and these might be harder to implement 
5 OnAction also is specific to Ul input OnOver or OnOut for instance, Imply some fomi of 

mouse, JdysUck or pointing device (as opposed to" touch panel). 

OnAction capabilities are not required. They are intended for devices that support compound 
requests and are likely to run browsers, eto. 

OnAction works atong with other functionality like ChangeVariable etc. to bring about scripting 
10 action. 

- ERRORS . 

VPINVAL - invalid arguments 
VPINVA^ORAWABLE • invalid drewable 
IS VPi^OTSUPP- The procedure Is not supported 

VPNOWIDGETS - OnAction is supported but not for Widgets 
VPT00CC»4PLEX - The Onaction request was too complfcated 

5.69 ChangeVariaWe 
20 SYNOPSIS 

ChangeVariable □ Vpstat 

ARGUK^ENT 

Enums vamamefype { 
25 SHOWFROM=0. 
COL=1. 
TEXT=2, 
ISITALIC =3. 
UNDERUNE = 4. 
30 COL2 = 3 

}: 

Union newval switch (vamametype vnt) { 
CaseSHOWFROIi/l: 
35 Int newshowfifom; 

Case COL 

Col newCol; 
Case TEXT: 
SGing 
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Case ISITALIC: 

Void; 
Case UNDERLINE: 

Void; 
Case C0L2: 

Col newCol; 
};" ' " ' " 

Enums reflype { 
WID i= 0, 
REQNUMsl, 

}: 

Union Objref switeh (refiype rtype) { 
Case WID: 

- Wid widref; 
Case REQNUM: 

Unsigned short reqref; 

}: 

Staict ChangeVariableatgs { 
Drawable drw; 
Objref ore^ 
Newval value; 

}: 



RESULT 
VPStat; 

DESCRIPTION 

ChangeVariable is used as part of OnAction Operations, it is intended to support changes for 
OnAction operations. Although It maybe possible to have OnAdion, support for ChangeVariable is 
reconvnended If OnAction is implemented. 

This procedure is not mandatory. 

IMPLEMENTATION 

ChangeVariable alters Ul state. For Instance showfrom on a Te>ctBox will change the visible 

contentwtHesorielf. ~ ' " 
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ERRORS 

VPINVAL - Invalid arguments 
VPINVALDRAWABLE - Invalid dravrable 
5 VPNOTSUPP- The procedure is not supported 

VPNOWIDGETS.- OnAction is.supported but notfor Widgets 

5.70 CreateVisualObject 
SYNOPSIS 

10 CreateVisualObjedArgs DObjectU 

ARGUMENT 

union VP_args swHcli (procnums proc) { 
Case <OPCODE>: <argument>; 

15 

> 

stnict Compoundargs { 
VP_args VP_argarray<>; 

20 }: 

Union AlptiaB switci) (boots isAipiia) { 
Case TRUE: 

Stiort alpha; 
25 Case FALSE: 

Void; . 

} 

Stmct CreateVisualObject { 
30 Drawable drw; 

Bools isVisible; 
Coordinate Location; 
Alphas alphaBlend; 
stmct Compoundargs cpa; 

35 } 

RESULT 

ObjectU; 
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DESCRIPTION 

A Visual Object is a group of drawing requests that represent an object The Visual Object can 
be created shown and moved. These could include cursors, animations, etc. 

CreateVtsualObject creates a Visual Object The Compound Requests are expected to be 
simple actions and should not contain OnAction primitives. They must also all occur on the one 
drawable. The Compound Requests occur relative to the Location (they assume that Location is the 
top left comer). 

If isVisible is FALSE, the Object is Invisible when created. If isalpha is set (on systems that 
support ft), then the Object is Alpha Blended into the screen. 

IMPLEMENTATION 

VPCIients that can utilize this capability are higher end devices. .These are typically deyioes 
that have sophisticated Ills or are graphically Intensive systems like Gaming Consoles, animation 
enabled AV reading and entertainment devices. 

Cane must be taken to ensure that the system can move the visual object intoto against a 
background etc. More capabilities along these lines will be available in ftjture versions of the 
protocol. 

VP Client implementations must strive to completely support VisualObject features or not at 

all. 

ERRORS 

VPINVAL - invalid arguments 
VPINVALDRAWABLE - invalid drawable 

VPINVALNOALPHA - Alpha blending is not supported on the device 
VPNOTSUPR - The procedure is not supported 

5.71 ShoWVisualObject 
SYNOPSIS 

ShowVisuatObjectargs OVPstat 

ARGUMENT 

Union alphaU switch (bools selAipha) { 
Case TRUE: 

Short alpha; 
Case FALSE: 
Void; 

■} . ' ' 

StmctShowVfsualObjec(args{ ^ " 
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Objid object; 
Coordinate Location; 
AlphaU au; 

} 

5 

, RESULT.. : - - . - 

Vpstat; 

DESCRIPTION 

10 The Shovt/Visual Objert primitive is used to display a visual object to the screen. It Is a simple 

primitive used to bring a Visual Object onto the screen. If alphaU Is set. then the alpha blending 
value is also set _ _ , 

IMPLEMENTATION 
IS The Object is brought onto the screen at the specified location. 

ERRORS 

VPINVAL - invalid arguments 
VPINVALDRAWABLE - invalid drawable 
20 VPINVALOBJECT- invalid Object 

VPINVALNOALPHA - Alpha blending is not supported on the device 
VPNOTSUPP - The procedure is not supported 

5.72 HideVisiialObject 
25 SYNOPSIS 

Objid □ Vpstat 

ARGUMENT 
Objid 

30 

RESULT 
Vpstat; 

DESCRIPTION 

35 The aim of HideVisualObJect is to hide the object 

IMPLEMENTATION 

The Implementation requires that the object be hidden from view. 
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ERRORS 
- VPINVAL - invalid arguments 

VPINVALDRAWABLE - invalid dtawabie 
S VPINVALOBJECT- invalid Object 

_ . . - VPINVAU40ALPHA- Alpha blending is not supported on the device . - 

VPNOTSUPP - The procedure is not supported 

5.73 MoveVisualObject 
10 SYNOPSIS 

MoveV/isualObjectaigs OVpstat 

" ARGUMEIsif 

Struct MoveVisualObjectargs { 
15 Objid object; 

Location where; 

} 

RESULT 
20 Vpstat; 

DESCRIPTION 

The IVloveVisualObject request requires that tiie Visual Object be moved to the specified 
location. 

25 

IMPLEI^ENTATION 

ERRORS 

VPINVAL - invalid arguments 
30 VPINVALDRAWABLE - invalid diawable 

VPINVALOBJECT -invalid Object 

VPINVALNOALPHA - Alpha blending is not supported on the device 

5.74 ResizeVisualObject 
35 SYNOPSIS 

ResizeVisualObjectargs □ Vpstat 

ARGUIWENT ' " 
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Struct ResizeVisualObjectaiigs { 
short WidthScale.- 
shortHeigh^le; 

}: 

5 

RESULT. . - - . - - - 

Vpstat; 

DESCRIPTION 

10 This resizes the visual object's Width and Height. The Width and Height are on a percentage 

scale. (200 doubles and 50 halves, while 1 00 leaves it as Is). 

t 

IMPLEIWENTATION 

These are operations for more advanced graphical displays. 

15 - 

ERRORS 

VPINVAL - invalid arguments 

VPINVALDRAWABLE - invalid drawable 

VPINVALOBJECT - invalid Object 
20 VPINVALNOALPHA - Alpha blending is not supported on the device 

5.75 Pause 
SYNOPSIS 

Short -> Vpstat 

ARGUMENT 
Short; 

RESULT 
30 Vpslat; 

DESCRIPTION 

Pause is used in Compound requests to simulate .slow movement Pause values are in 100s 
of microseconds. Thus Pause 100 pauses for 10 million seconds. 

It is recomrnended that Pause operations be for very short duration. 

IMPLEMENTATION 

The Pause should be carefully implemented. Long pauses might not be in the 
Interest of-the handsetrVP-Client8-<aiire3?srci§e^flie liberty to avoid very tong pauses. 
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ERRORS 

VPINVAL - Invalid argumente 

VPTOOLONG - the pause request is for too long 

5 

. . .5.7jB TTSRequest _ . 

SYNOPSIS 

String □ Vpstat 

10 ARGUMENT 
String 

RESULT 
Vpstat 

DESCRIPTION 

The TTS Request causes the request to convert the string to audio and play It TTS request is 
converted to speech and played using a voice supported by the VP Client and available on the client 
device. 

IMPLEMENTATION 

There are multiple possibilities. FestivalI14] is a commonly used and publicly available 
TextToSpeech system. Multiple other systems exist and can be developed. 

25 

ERRORS 

VPNOTSUPP - The VP Client does not support TTS 

6.77 SpeechStneam 
30 SYNOPSIS 

. SpeechStreaniargs □ Vpstat 

ARGUMENT 

Stnict SpeechStreamaigs { 
35 bools incliannel; 

SpeechDataFomiat dataforma^ 

opaque bytesofaata<MAXSPEECHDATA>; 

} 



15 



20 
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RESULT 
Vpstat; 

DESCRIPTION 

5 This is used to play the appropriate raw speech that is sent Section 9 outlines the various 

formats that could be negotiated. The VP Client uses thls for Inband speech streams using the RPC 
form of communication. 

If it is out of band and Inchannel is FALSE, then the format alone is specified. This Intimates 
the client device about the contents. 

10 

IMPLEMENTATION 

This is an alternative to for TTS and for multimodal communication. VP djents shouId.be able 
to iinplemeht atieast one of the fomiats specified and servers must provide data in that forniat This 
is not to be used for Media Streaming, (in this case separate media streaming technologies lilce RTP 
15 sen^e better). 

Certain applications may use this for Telephony API implementations and applications. 
ERRORS 

VPNOTSUPP - the procedure is not supported 
20 VPINVAL - the arguments specified are invalid 

VPFORIWATNOTSUPP - the specified format is not supported 

5.78 VoiceCall 
SYNOPSIS 
25 PhoneNumber □ VPstat 

ARGUMENT 

Struct PhoneNumber { 
String phonenum; 

30 } 

RESULT 
Vpstat 

35 DESCRIPTION 

This specifies the phone number to call. Thl$ is used to support Clidc-To-Voice functionality to 
enable Computer Telephony Integration and similar capability. 

liWIPLEMENTATION ' 

-118- 



wo 01/91482 



PCT/USOl/17274 



The String specifies a phone number. The string should be strictly numeric and DTMF. 
This may not be supported for some applications as the user may disable availabilify to this 
for certain classes of applications. 

VP Clients could maice this configurable on the client device (or sen/ers could do this using a 
s user profile setting). 

ERRORS 

VPNOTSUPP - there is no support for this feature 
VPINVAL - the phone number string was invalid 
10 VPPERMISSIONDENIED - the application was denied pemnission to do Click-To-Voioe 

5.79 Notification 
SYNOPSIS 

Void 0 NotrficationRes 

15 

ARGUMENT 
Void 

RESULT 

20 Enums NotificationRes { 

WILLRECONNECT = 0, 
BUSY=1, 
DISABLED = 2, 
DONTCARE = 3 

25 } 



DESCRIPTION 

This is used for asynchronous notifications thus enabling servers to inform clients of events 
and thus call back. The intention is to allow applications like MMS to provkle immediate notification 
30 to clients about Impending events lite messages, ete. 

The VP Client either intemipts its current application and reconnects, indicates that it is busy, 
or that the notiftoatlon feature Is temporarily disabled or that it does not care about it 

IMPLEMENTATION 

35 A VP Client that wishes to implement this feature, must be able to wake up on a sender skle 

transmission, check to see if actton is necessary and respond to It 

A session even in the suspended state could be reinvigorated using this feature. 

ERRORS : 
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VPNOTSUPP - there is no support for this featuiB 

6 The Event System Protocx)! 

The Event System Protocol is mainly used to allow handsets to send back User Driven events 
back to the sen/er. The vartous Jnteradfons require that user Events are delivered back to the VP 
server. The VP Client Implementatton requires that Widget Events. Clicks, Speech Events, etc. be 
delivered back to the server. There is a lot of interadion that Is spedfic to the client device. 

The-Event System Protocol from the RPC point of view woite with the client device being the 
RPC client and the VP Sender being the RPC server. 

The Event System is Initialized at the behest of the VP Server when it issues an 
InlBallzeEventSystem call. This leads to the VP client identifying ail supported Event Systems. The 
. specific the port pn the VP Sender that the dient should use for the Event System - 

Connection, (where the Event System connection resides). . 

It is possibte for the VP Client to use an agency for the Event System Prolocpl. If this is the 
case then the VP Client sends ite 

The client sends an InitEventSystem request and infonnation on each of the input capabilities 
available on the device. 

FreeEventSystem is typically done when the Event System Is freed. 
DellverNextEvent delivers another event to the VP Server. 
DeliverEvents delivers a sequence of events to tiie VP Sender 

6.1 Event Types and Systems 

Most Event Systems are grouped under one of the following - Keyboard. Mouse, Joystick, Touch 
Panel, Remote or Speech Driven. 

There could be otiier variations that could simulate one of tiiese event systems. 
There are multiple types of requests tiiat tiie event system could generate. They could be simple or 
tiiey could be compound requests. Simple requests are requests like a Click or a Touch Event. The 
Remote is used to specify tiiat tiie use case where there are a few buttons on the device that are 
specially mapped by ttie application and ttie Ul system (UlSPEClFIC) (in this case the buttons are 
mapped in order on ttie handset from 1 at tiie top left moving horizontelly till the end of tiie list - a 
later draft release could address a button mapping system). Compound requests are a series of 
clicks and moves. A line drawn on a touch panel couM be a Cohipound Request 
Keyboard typing couW lead to a character string being sent back to the server 

6.1.1 Consts and Enums 
Const MAXHOSTID = 20; 



Enums evtsystype { 
— MOUSE = 1i 
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TOUCH = 2, 

KBD = 3. 

REMOTE = 4. 
SPEECH=5, 
JOYSTICK* 6 

}; 

enums evtiype { 
KEYPRESS = 1, 
KEYLIST = 2. 
TOUCH = 3. 
BUTT0NPRESS = 4, 
BUTT0NRELEASE = 5. 
.MOTIONNOTIFy = 6, 
CLICK = 7, 
SELECT = 8, 
SCROLL = 9, 
SPEECH = 10, 

}: 

struct keypos { 
short x; 
shorty; 
short key; 

}; 

enums datatype { . 
ASCII = 0, 
UTF16=1, . 
DTMF = 2. 
UISPECIFIC = 3 

): 

Union Butposdata switeh (datatype dt) { 
Case ASCII: 

Charlypedchar; 

Case UTF16: 

utf16 utf16chan 

CasrOTMFr 
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Case UlSPECIFIC: 

Char UlSpecificKey; 

}; 



Union Keylistdata switch (datatype dt) { 
Case ASCII: 
String data; 
10 Case UTF16: 

Utf16iJt«6data<>; 

Case DTMF: 
Chardtmldatao; 
15 Case UlSPECIFIC: 

Char ufsdatao; 

}; 

struct keylist{ 
short x; 
20 shorty; 

Keylistdata kdata; 

}; 

struct tpos{ 
25 . short x; 

shorty; 

}: 

struct butpos{ 
30 Butposdata bpdata; 

}; 

struct motion { 
tposfrom; 
35 tposto; 

}; 

struct scroliin1b{ 
Wid which; 
BooIsUP; 
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} 

union Event switch (evttype etype) { 
case KEYPRESS: 

keypos kps; 
caseKEVUST: 
- -keylistkfs; ----- 
case TOUCH: 

tpostps;. 
case BUTTONPRESS: 

butpos bps; 
caseBUTTONRELEASE: 

butpos bps; 
caseMOTIONNOTIFY: 

nnotton mps; > 
case CLICK: 

Wid ws; 
case SELECT: 

Wid ws; 
case SCROLL: 

scrollinfb si; 
case SPEECH: 

}; 

struct NextEvent{ 
int EventOlD; 
int EventSeqNum; 
Event evt; 

}: 

enumskbdtype{ 
ENGQWERTY = 1. 
KAN Jl = 2, 
DTMFs3, 
OTHER = 4 

}: 

struct keystruct{ 
Int EventOlO; 
Icbdtypg'l^rpe; 
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}; 

enums mousetyp6{ 
TW0BUTT0N = 1, 
THREEBUTTON = 2 

}: 

struct mousestruct{ 
intEventDID; 
mousetype mtype ; 

}; 

enums fouch^pe{ 
NORMAL =1. 
COMPOUND = 2 

}: 

struct touchstruct{ 
intEventDID; 
touchtype ttype; 

}: 

struct speechstruct { 
Int EventDID; 

speechfomiatdata sfd; /* see section 9 */ 

} 

union EventSystem switch (evtsystem systype) { 
case MOUSE: 

mousestruct ms; 
case TOUCH: 

touchstruct ts; 
case KBD: 

keystruct ks; 
case REIWOTE: 

keystruct krs; 
case SPEECH: 

speechstruct 8s; 
case JOYSTICK: 

mousestruct nnjs; 
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enums hosttype { 
INET=1, 
GPRS = 2. 
OTHER 

5 }: 

. t^)edef enums hosttype hosttype; ..... 

union Host switch (hosttype hstype) { 

case INET: 

charinetaddr[4]; 
case GPRS: 

chargprsaddrfS]; 
case OTHER 
char <>; 

}: 

struct EventSystemlnfo { 
Host hostid; 
Opaque hostcoolcie[4]; 
EventSystem evtsyso; 

} 

struct PhoneNum { 
String dialnum; 

}: 

25 6.2 Speech as Input Events 

Speech also functions to serve as an input mechanism. There are several ways to send 
speech input There are both inband and out of band methods of sending speech inputs from VP * 
Clients. Typically in mostly data environments where speech brings a multimodal capability, the use 
of speech is very periodic and bursty. Thus a phone call dedicated to this mode might or might not 

30 be economically appropriate and the right approach from a resource usage point of view. It is 
however important to recognize Vnal certain liandsets might use this. 

Speech can also bie used out of band (with respect to this protocol) but through the data 
channel. In this case, a separate channel is established. The VP Client device recommends the use 
of the contemporary Session Initiatidn Protocol for this purpose. (rf<2543). 

35 

6.2.1 Out of Band Speech Events 

As previously indicated, Out of Band Speech could use two fonms of communication. There is 
the speech call that causes the client to dial a number. If the server supports the use of a phone call 
for speechrandhaFnot already Indiratedtfi 
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Uie Event System Protocol to obtain the remote number (using GetPhoneNum). This fbmi of 
Interaction Is feasible for some classes of devices (like GPRS Class A devices) which support 
slmuHaneous voice and data connectivity. 

The use of SIP (or H.323) malces it possible to liave a remote server that is waiting for l^ultiple 
5 speech fonnats are supported by VP protocol. The presence of many fomiats serves to allow VP 
. clients to send speech data in any one of-the negotiated fonnals. However, the client must find the 
server and port to connect to, the speech fomiat and negotiate the connection successfully. The 
Event System Protocol ensures this using the Ge^peechSeiverlnfo request 

It is also necessary to stete that additional server to sen/er protocols will be necessary to 
10 esteblish the right movement of speech data and infonnation bacic to the VP server mnning the 
multimodal application, (this lies beyond the scope of this specification). 

6!2.2 in band Delivery of Speech Events 

The Inband delivery of speech is to use DellveriMextEvent to send speech date back to the 
15 server. The Inband delivery of speech implies that the server should by itself support the needed 

recognition and capabilities, and if it did not should seek and find the agency necessary and 
communicate with it. 

This makes the delivery of inband speech events simpler. 

20 6.3 InitEventSystem 
SYNOPSIS 

EventSystemlnfo o EVstet 

ARGUMENTS 

25 Union SpeechServerlnfo switch ( enums Speechfonnat) { 

CaseSPEECH^CALL: 

String PhoneNum; 

Case SPEECH_SIP_RAW, SPEECH_SIP_RPE_LTP, SPEECH_SIP_SPHINX: 
Address SIPSenrer; 
30 Case SPEECH_EVT_«AW. SPEECH_EVT_RPE_LTP, SPEECH^EVT_SPHINX: 

Short hostcookie; 

} 

Struct EventSystemlnfo { 
AppilcatibnidAld; 
35 Host Hostid;. 

Opaque hostcookie[4]; 
EventSystem evtsyso; 
} . 
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RESULTS 

Struct Evstatlistf 
Evstatstato; 

} 

5 

DESCRIPTION 

The EventSystemlnfo request specifies the various event systems on the VP client that are to 
be initialized. The Hostidyhostcookie pair is either the VP server itself that was indicated in the 
Graphics System Protocol or the agency cunently being used by the client. 
10 The reply specifies whether the VP Server was satisfied with the request or not 

IMPLEIWIENTATION 

The speech server could be the VP Server's recommendation or flie server that the VP Client 
cunently uses. If the Client uses a PhoneNumber already and does not wish to change, the VP 
15 sender could expect a SIP connection that originates at the other end of the phonecall. If the Client 
uses an existing SIP connection and does not wish to change, the Client specifies a 
Hostid/Hostcookie to the sen/er from where the VP server can expect to receive its speech 
infomfiation. 

Additionally, an agency or proxy might be used. If this is the case then the client must specify 
20 the peer's address and port to the VP server (through the agent or proxy). Further interactions with 
the proxy must also prepend Application Identifiers to identity the application. The server will then 
contact the peer host to receive information (before it replies). The VP Client then continues to send 
future DeliverEvents and DeliverNextEvent request through the proxy to the VP sen/er. 

25 ERRORS 

EVBADSERVER - Specifies that a server Is bad 

6.4 FreeEventSystem 
30 SYNOPSIS 

EvenSystemlnfb □ void 

ARGUMENTS 

EventSystemlnlb (as above) 

35 . 
RESULTS 
Void; 

DESCRIPTION - 
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This is used to free event system and to release connections. It releases tied down or 
dedicated resources inside the network. 

IMPLEMENTATION 

The release of resources must be done by the VP Client as well if it uses additional resources 
(disconnecting phone calls, SIP connections etc.) - - - - 

ERRORS 

6.5 DeliverNextEvent 
SYNOPSIS 

NextEvent □ Evstat 



ARGUMENTS 

Struct NextEvent { 
ishort EventDID; 
short EventSeqNum; 
Dnawable dtvr. 
Event evt 

}: 

RESULTS 
Evstat; 

DESCRIPTION 

The NextEvent is delivered to the server in sequence (with the SeqNum incremented on each 
request). This is the most common request sent to the server. When it includes a speech event sent 
inband, it is a simple padtet with the fomiat indicated (see Section 9). 

IMPLEMENTATION 

The Implementation requires that the Event System requeste be sent in sequence. 
If delivered through an agency or proy, the VP Client sends the Application ID as the first 
parameter 

Of the request The agency or proxy uses this to detennine where to send the request to. 
ERRORS 

EVOUTOFSEQ - the Event was delivered out of sequence 
EVSPEECHDATATOOLONG - the Speech data was too long 
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6.6 IDeliverEvents 
SYNOPSIS 

OeliverEventsList □ Evstatlist 

S ARGUMENTS 

Struct SeqEvent{ 

short EventSeqNum; 
Event evt; 

}; 

10 struct DeliveiCventsUst { 

short EventDID; 
Drawable drw; 
SeqEvent Eventlisto; 

}: 

15 - 

RESULTS 

Struct Evstatlist { 
Evstatstato; 

} 

20 

DESCRIPTION 

Delivers a sequence of Events intended for the Application. The aim of this is to send multiple 
short events that are allied like touch panel line drawings, mouse movements, etc. The 
EventSeqNums conrespond and increase for the events. 

25 

IMPLEMENTATION 

The implementation requires that the Event System requests be sent in sequence. 
If delivered through an agency or pre)xy, the VP Client sends the Application ID as the firet 
parameter of the request The agency or proxy uses this to determine where to send the request to. 

30 

ERRORS 

6.7 SelecUtem 
SYNOPSIS 
35 Selectltemaigs □ EVslat 

ARGUMENTS 

- Struct Selectltemargs{ 
shiort EventDlDT 
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Drawable drw; 
Wid widget; 
short Hem; 

} 

5 

RESULTS. 
EVstat; 

DESCRIPTION 

10 The Seiectltem request allows the VP Client to send the selection of an item t)y a client bade 

to the server. 

IMPLEMENTATION " ~ ~ ' 

This is possible on V for VP Clients that have support for Widgets. 

15 

ERRORS 

EVDONTKNOW - The server does not Itnow about ttiis Widget or Item 

6.8 ScroIIToText 
20 SYNOPSIS 

ScrollToTextargs O EVstat 

ARGUMENTS 

Struct ScrollToTextargs { 
25 Drawable drw; 

Wid widget; 
Short showfrom; 

} 

30 RESULTS 
EVstat; 

DESCRIPTION 

The ScrollToTexlargs enables the VP Client to indicate that the user scrolled down the page 
. 35 to a certain location. 

IMPLEMENTATION 

The Implementation specifies the area to showfrom. 
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ERRORS 

6.9 Reconnect 
SYNOPSIS 
5 Reconnectargs □ ReconnectresU 

ARGUMENTS 

Struct AcGveCachedDrawableList { 
Drawable dm<>; 

10 } 

Struct Reconnedangs { 

Appiicationid Aid; 
Host Hostid; 
Opaque hostoookie[4]; 
15 ActrveCachedDrawableUst acdl; 

} 

RESULTS 

Struct ReconnectEventSystemU { 
20 EventSystem evtsys; 

Short SeqNum; 

} 

Struct WillReplayDrawableList { 
Drawable dnwo; 
25 Short Vpseqnum; 

} 

Struct StaleCachedDrawableList { 
Drawable dnv<>; 

} 

30 Struct Reconnectres { 

ReconnectEven^stemU resU<>; 
WillReplayDrawableList wrdi; 
StaleCachedDrawableList scdl; 

} 

35 union Reconnec^resU switch (bools connected) { 

case TRUE: . 

Reconnectres nr; 
case FALSE: 

EVStet~s@t^ 
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}: 

DESCRIPTION 

The Reconnect request is sent by a VP Client to a VP sewer (possibly through an agent or 
proxy) In order to reconnect the VP Client baci< to the sen^r. As part of the reconnect request the 
VP Client indteaies thejist of Drawables. that It has cached Ibr this Application Instance. The Server - 
«ien Infonns ttie Client about the event systems It had In use, the sequence numbers both In the 
forward and reverse direction are reported as well. The Server tfien informs the VP Client that It will 
replay a list of drawables before it resumes. The Scdl spedfies the list of Drawables that the VP 
Client has cached that have gone stale, while the VP Client was disconnected. This is now 
discarded by the VP CHenL 

IMPLEMENTATION 

This request is used by the VP aient to reoonstmct its state entirely. As the client relies 
entirely on the networl< to reconnect itself, the network must cache all infomiation needed and the 
VP Client must work with the server to reestablish its state. 

ERRORS 

6.10 StateTransitionEvent 
SYNOPSIS 

StateTransitionEventargs □ EVstat 

ARGUMENTS 

Stmct StateTransitionEventargs ( 
SessionState old; 
SessionState new; 

} 

RESULTS 
EVstat; 

DESCRIPTION 

This is used to change state by the VP Cfient The new state setting results in actions that 
lead to a change In the state. Typically the VP Client initiates the change in the underlying session 
and connection state after receiving an OK reply. 

IMPLEMENTATION 
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It is important for the VP Client to initiate the state change. The VP Client after initiating the 
state changemust continue to handle actions if notifications are to be expected finom the client 

ERRORS 

5 VPir«IVAL - arguments are invalid 



6.11 SpeechServerinfo 
SYNOPSIS 
ARGUMENTS 
10 RESULTS 

DESCRIPTION 
IMPLEMENTATION 
ERRORS 

15 7 The Application List Protocol 

In the service environment of the VP client, multiple servers can exist The VP client 
bootstraps itself back to its previous state using the Application List Protocol. The VP dient also 
migrates between the various applications running on various VP servers in the sen/ice environment 
using the Application List Pnatocol. 

20 The Application List Protocol participates in the bootstrap process to enable clients to return to 

their previously existing state. The VP Client using DHCP finds the host and port where tiie ALP 
resides and Is running. It then starts eitiier the ALM Ul application or reconnects back to its 
previously mnning application on the VP server (getCurrent Application) or it could connect to any of 
the Users cun^ntiy running applications (through tiie use of eitiier tiie ALM Ul or a native Ul) or it 

25 could Launch a new Application. 

7.1 Types and sbuctures 
typedef short Appid; 

30 typedef string LocationURL; 

enums ProgranriEnvType { 
PJAVA = 0, 
PJAVAEXT = 1. 
35 MIDPsi, 

CLDC = 2, 
CL1_P = 3, 
CLI_C = 4 
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enuinsFeatureSet{ 
VISUAL = 0. 

SPEECHINPUTENABLED = 1. 
SPEECHINPUTREQUIRED « 2, 
SPEECHOUTPUTENABLED = 3, 
SPEECH0UTPUTREQUIRED = 4, . . 
MULTIMODAL = 5, 
AUDI0ENABLED = 6, 
AUDI0REQIURED = 7, 
ViDE0ENABLED»8, 
VIDEOREQUIRED - 9 

}; 

enums Launcher { 
VPCLIENT = 0. 
ALM = 1, 
ALMUl = 2. 
APPLICATION 

}; 

struct Program { 

CiassmarkU CU; 
short Environment; 
Featureset featureso; 
Locat'onURL programURL; 

}; 

struct Sen/er{ 

Address Vphost; 

} 

union LaunchedBy sM^'tch (enums Launcher) { 
caseVPCLIENT: 

void; 
caseALM: 

Server ALMhost; 
case ALMUl: 

Server ALMhost; 
case APPLICATION: 

Applicationlnstanoe LauncherAI;' 

}; 

struct Applicationlnstanoe { 

^Appld— aid; 
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Server host; 
opaque hostcookie[41; 
Program program; 
LaunchedBy parent; 

5 }; 



11 GetCurrent^plication 
SYNOPSIS 

. VoidDApplU 

10 

ARGUMENTS 
Void; 

RESULTS 

15 union ApplU switch (ALMstat stat) { 

caseALMOK: 
Applicationlnstance ai; 

20 case ALMUILAUNCHED: 

Applicationlnstance almuiai; 

default 
void; 
25 }; 

DESCRIPTION 

This returns the cun-ent Application Instance being run by the VP Client. The VP Client 
bootstraps itself to continue mnning the application it was running before it was disconnected or 
30 turned off. The Applicationlnstance is a sender/port pair where a VP Server exists. This allows the 
client to connect back to the Application Instance. 

IMPLEMENTATION 

The VP Client uses this to detemiine its current application at power On or following a 
35 disconnection with Its environment The ALM could sometimes laundi the ALM Ul application if the 
VP Client is configured to require such a launch and return that as the cun-ent application. This Is 
possibly for a first time poweron. Typically, the ALM returns the VP Client to ite last mnning 
application before it was powered off or disconnected. The ALM could use additional server to 
sender protocols to obtainlnformatiiSnWdut^^ 
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document). The VP Client would then do a RECONNECT to the Application Instance, retrieve Its 
prior state and proceed from there on. 

ERRORS 

5 ALIMDATABASEERROR - could not retrieve state Info for Vne VP Clfent 

ALMPERMiSSIONDENIED - the retrieval vi^s not possible as the ALM was d 
permis^n 

7.3 GetCumentApplicafionUst 
10 SYNOPSIS 

Void □ ApplUstU 

ARGUMENTS " " " 
Void; 

15 

RESULTS 

union ApplListU switch (ALMstat stat) { 
case ALMOK: 

Applicationlnstance ai<>; 

20 default 

void; 

}; 

DESCRIPTION 

25 This returns the list of all Applications that the VP Client is currently running with infomiation 

on aO of them. 

IMPLEIWENTATION 

The request is made to find all applications the VP Client is cunently mnning. This could be 
30 used to find a particular application or to connect back to an application the user wants to use. A 
native application on the VP Client could also use this to list the list of VP Applications that the user 
has ainnlng inside the networtc 

The request implies the detemiination of pemfiissions ete. through a sen/er to sender protocol 
and the server discovery mechanism is beyond the scope of this specification. However it must be 
35 said that the VP Client might have multiple air interiiaces (WLAN vs. Wide Area) and could be 
switching from one to the other. VP Clients might still be able to discover and connect to the 
applications they were previously running. 
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ERRORS 

ALMDATABASEERROR - could not retrieve state info for the VP Client 
ALMPERMISSIONDENIED - the retrieval was not possible as the ALM was denied 
permission 

5 

7.4 LaunchApplicatlonManager - * - . - . . , . . 
SYNOPSIS 

Void □ LaunchU 

10 ARGUMENTS 
Void; 

RESULTS 

union LaunchU switch (ALMstat stat) { 
15 caseALMOK: 

Applicatlonlnstance al; 

default 

void; 

}: 

20 

DESCRIPTION 

This is used by the VP Client to launch an ALM Ul application; 
IMPLEMENTATION 

25 The VP Client could be configured to not have the ALM automatically launch the ALM Ul. The 

VP Client could do so when it chooses thus saving resources. 

ERRORS 

ALMPERMISSIONDENIED - could not launch as pemiission was denied 

30 

7.5 NewApplication 
SYNOPSIS 

NewApplicationaigs □ LaunchU 

35 ARGUMENTS 

Union LaunchOn switch (bools specify) { 
Case TRUE: 

Server serverllsto; 
~ CMe"FACSEl 
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Void; 

} 

Struct NewApplicationargs { 
Program newprog; 
LaunchOn whicServer; 

- } .... 

RESULTS 

union LaunchU switch (Al^stat stat) { 
ca8eALI\A0K: 

Applicationlnstanoe ai; 

deiault 

void; 

}; 

DESCRIPTION 

This request asks the ALM to launch a new application. The Launch requires the application 
to be launched on a sender. A series of servers could be specified by the VP client or it could be left 
empty implying that the ALM must find a server to launch the application on. 

IMPLEMENTATION 

This request launches a new application on an apprc>priate server. The ALM could use its 
capabilities to decide where It is to be launched. However the VP Client can refuse to connect if it 
does not like the server that it is actually launched on and request termination. 

ERRORS 

ALMPERMISSIONDENIED - could not launch as pennission was denied 

7.6 GetAvallableServer 
SYNOPSIS 

Program □ Sen/erU 

ARGUMENTS 
Program; 

RESULTS 

Union ServerU switch (ALMstatstat) { 
Case TRUE: 

Sen/er -sen^erllsto; ' 
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Case FALSE: 
Void; 



} 



5 DESCRIPTION 

- - This request allows the VP CHent to determine a list (rf available servers that are ca^^^ 
launching the application on. This is a sirnple request that allows the VP Client to find a sender in the 
neighborhood (potentially new) and to request a launch of an application on it. 



10 IMPLEMENTATION 

The VP Client could choose to launch the application on any of the servers in the list possibly 
choosing an appropriate one. The request irnpjies that securi^ verification, environment 
compatibility, etc. have been done. 

15 ERRORS 

7.7 GetAvallableProgramList 
SYNOPSIS 

GetAvailableProgramListU □ ProgramListU 



20 



ARGUMENTS 



Enums iistsspecified { 
LISTGROUPS = 0, 



25 



LISTF0RAGR0UP = 1. 
LISTF0RGR0UPS = 2, 



ALLPROGRAMSsS 

}; 



Union GetAvaiiableProgFamListU switch (Iistsspecified Is) { 
Case USTGROUPS: 



30 



Void; 

Case LISTF0RA6R0UP: 
String LIstGroup; 

Case USTFORGROUPS:. 
String ListGroupso; 

Case ALLPROGRAMS: 
Void; 



35 



}; 



RESULTS 
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Union ProgramListU switch (ALIMStat stat) { 
Ca8eALM3K- 

Program programllsto; 
CaseALMLISTS: 
s String ListGroupso; 

- - Default - - - . . _ 
Void; 

} 



10 DESCRIPTION 

This is a request that provides the list of immediately available programs that are available In 
the vidnify. UsiGroups is used to send multiple groups available possibly from multiple sources. 

IMPLEMENTATION 

15 The VP Client uses ListGroups to explore its applications in its Virtual Neighboihood, The VP 

Client could be using a native application to obtain these lists. The ALMUl also could serve as a 
mechanism to access the same information and help the user browse the applications he/she can 
run. 

20 ERRORS 

ALMPERMISSIONDENIED - the ALM was denied pennissions to access infomiation 

7.8 SetCumentApplicationlnstance 
SYNOPSIS 
25 Aid OALMStat 

ARGUMENTS 
Aid; 

30 Results 

ALMStat; 
DESCRIPTION 

The request allows the VP Client to set the current application. This causes the ALM to infbm) 
35 tiie corresponding server of ah Impending RECONNECT and change its own database. 

IMPLEMENTATION 

The VP Client uses this to switch applications back to an older appHcafion. 
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ERRORS 

VPAIDINVAL • the Aid was invalid 

7.9 TerminateApplicationinstance 
5 SYNOPSIS 

-Aid □ ALI\flStat 

ARGUMENTS 
Aid 

10 

RESULTS 
_ AlJWStat 

DESCRIPTION 

IS - The request is used to terminate the application with the given Aid. It is intended to be a way 
by which the VP Client can terminate applications. 

IMPLEMENTATION 

This is used to terminate applications. The ALM will contact the VP Server and terminate the 
20 application and update its own database. The VP Client can then destroy all assodated caches and 
resources on the client device. 

ERRORS 

ALMPERMISSIONDENIED - The VP Client does not have pemiission to tenninate this 
25 application 

7.10 GetSecurifylnfo 
SYNOPSIS 
ARGUMENTS 

30 RESULTS 

DESCRIPTION 

IMPLEMENTATION 

ERRORS 

35 8. The Speech Data Fomriat 

A variety of fomiats can-be used for speech transfer. Multiple compression schemes exist for 
speech and the resulting compression and speech quality varies. VP however does not specHy a 
format specifically for VOIP or schemes for handlirjg voice calls although it is entirely possible that 

-VP-eiient-Devices-might offer sachcapabilities: 
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8.1 Overviewof the Various Formats 

The Raw Speech Format represents speech In the raw forni. It is sampled at the negotiated 
rate and could use mu-law compression as negotiated. 
5 A variety of fonnats are possible. 3G PP has several defined fonnats for speech notably the 

- AMR Speech Fonmats defined In [5] and its references. The capability of handsets to negotiate for . 
the most appropriate fomnat for Voice and multimodal applications is a useful and important 
requirement The GSM based fomnat is the most commonly used Wireless fbnnat for client devices. 
It Is important to note that client devices need to support only one of the fonnat If they are speech 
10 enabled, but could choose to support any number of them. 

The aim of the VP Client Implementation could be to choose the appropriate fomiat given the 

channel capabill^, noise, user dhoice. server capability, ete. so that there is the best possible 

speech recognition. 

15 8.2 Raw Speech Fonmat 

The Raw Speech Fomrjat involves sampling speech data at a specified rate (usually between 
4000 and 16000 times a second). It is often the case that such data could be companded using a 
mu law quantizer. 

Further it is possible that the codec could use one, two or even four bytes to sample the 
20 speech. 

For this fomiat, thus the specification indicates three parameters 

StaictRawSpeech{ 

Short samplerate; 
25 Short bitspersample; 

Quantizer type; 

} 

8.3 ADPCM Speech Format 
30 In ADPCM or adaptive differential pulse code modulation, speech samples are coded with the 

step size adaptively adjusted (based on the speech samples) and a predictor that is adapfive. The 
resulting speech i^ compressed and for the same bit rate of higher quality. 

Struct ADPCM { 
35 Short samplerate; 

Quantizer type; 
Short PredictionOrder; 

}; 
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8.4 GSM Speech Format 

GSM uses the RPE-LTP scheme for speech compression. This uses both short and long term 
prediction and additionai channel coding for recovery in en-or prone channels. The GSM speech 
format is a logical choice given the presence of a large number of existing handsets and the 
5 ovenA^elming preponderance of GSM. Multiple rates are possible with GSM although the client 
devloemightbecapable ofsupporting only one of them. - - ... 

VP Clients must ensure that they indicate the rate and class correctly. 

Struct GSM { 
10 Enums rate; 

Enums class; 

8.5 AMR Speech Fomiats 

15 AMR uses algebraic CELP. AMR is defined in [5] and is used by 3G PP for low bit rate speech 

coding. It defines 8 different biodc sizes for its MR-ACELP scheme. For speech recognition, the best 
possible speech fomiat which is probably the highest rate class (12.2 l<bps) is best chosen. It 
defines several modes for AMR speech. 

20 Struct AMR { 

Enums encodedblocksize; 
Enums mode; 

} 

25 8.6 Sphinx Speech Fonnat 

The Sphinx Speech recognition system was developed by CMU and uses homomorphic 
speech analysts for speech recognition. Delta Cepstral coefficients are used in Sphinx. This fonnat 
is also added to enable VP Clients to send tiie Sphinx input fonnat preceded at the client device. 

30 8.7 Silence Detection and Removal 

Silence Removal and detection could use the AMR - VAD schemes. Silence lengths must be 
specified as part of the speech code format 

StnictSilenoeLength{ 
35 int milliseconds; 

}. 

8.8 Preprocessing of Speech for Noise Removal 
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Discussions relating to Noise removai are currentiy deferred in this draft They will be added in 
a subsequent draft if appropriate. 

9. IMultimedia Support 

i\/iuitimedia support is required tif various applications and its capabilities are needed in a 
.variety of different ways. Multimedia is however an optional feature in VP Clients and it is intended 
to support additional stream management and software control to content, its streaming and the 
running of appitications in the MExE service environment that relate to streaming. 

Support for delivery (and management) of multimedia content on the client device is provided 
via the Delivery l\4ultimedia integration Framewori< (DMIF) which is a part of the MPEG-4 standard. 
The primary objective in using this frameworic is to enable maximum reuse of existing standards and 
protocols, and to prevent dupllcaton of eflbrt and functionality. The DMIF specification provides an 
architecture that is immune to fijture changes/enhancements in delivery technology, and protects 
investments made in multimedia temninal/applicatlon development. Further, the DMIF architecture 
has been demonstrated in implementation under the MPEG-4 standard. 

The DMIF specification is a subset of the MPEG-4 specification, and deals with the delivery 
layer of the MPEG-4 standard. The DMIF architecture, which includes the DMIF Application 
Interface (DAI), the DMIF layer, and the DMIF Networi< Interiiace (DNI), provides a delivery 
technology-independent mechanism for real time distribution of multimedia content MPEG-4, by 
design, targets multiple operating sihjations such as local retrieval, remote interaction, multicast etc., 
and multipte delivery technologies. DMIF mal<es this possible by providing transparent access to 
content, using URL addressing schemes, without regard to the underiying delivery technology used. 

The MPEG-4 client application ninning on the VP Sen/er is configured to decode and present 
(selectively) any or all of the streams (audio & video) obteined from the origin server. The origin 
server makes use of the DMIF architeclure to pacltage content for distribution over a variety of 
transport channels. The DAI , the DMIF layer and the DNI enable this capability in a media- 
independent and transport mechanism-independent fashion. Further the VP Server can choose to 
maintain over-the-air stream rate within bounds and maximize the quality for a given stream rate. Or 
it can choose to minimize the stream rate for a given picture quality. The VP server with a controlling 
application (possibly written using MPEG-J) can be configured by the user to manipulate and control 
streams to best isuite his own needs. Further it can peribmi the necessary rate adaptebon and error 
control that only an IntenAening edge located entity can. 

9.1 Delivery Layer Abstraction 

The delivery layer in DMIF consists of a two-teyer multiplexing sdnewe and manages the 
synchronized delivery of streaming infonnation from source to destination utilizing QoS features as 
available from the networi^. The first multiplexing layer enables grouping of elementary (audio & 
video) streams with low multiplexing overtiead. This is useful, for instance, for grouping streams with 
r-QoS-requirements.-This layer iS"speclfiea"in^art-6 of 
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multiplexing layer — the transport multiplexing layer ~ is not specified in MPEG-4. However, the 
interface to this layer is specified in the standard, and allows the transport of MPEG-4 content over a 
variety of transport mechanisms such as RTF (UDP, IP), ATM ML2, H.223 (PSTN), DAB etc. The 
choice of the particular transport scheme is left to the service provider. 
5 The DMIF application interface sits between the synchronization layer and the DMIF layer. 

The synchronlziation layer manages the Identification and synchronlzaHon (through fime-stamping) 
of elementary streams of different types (scene descriptors, video frames, sprites, audio eto.).The 
DMIF network interiace sits between the DMIF layer and the transport layer, and provides a uniform 
intertiace to the upper layers which are unaware of the transport mechanism. 

10 

9.2 Client*€en^er Interaction 

- The DMIF nwdules at the-dlent and server work together^ 

DMIF includes a signaling protocol — the DMIF Signaling Protocol — which facilitates interaction 
between a nemote terminal and a central server. The applications themselves are unaware of the 
15 signaling protocol. User interface commands, for example, are carried transparently by DMIF ftom 
the client to the server, and only interpreted In the application layer at the server; similariy. control 
messages from the server are passed as opaque data to the client where the application layer takes 
appropriate actions. 

For playback control of multimedia streams at the client end, the application sends standard 
20 user-interface commands such as PLAY, REWIND, and PAUSE to the server where the appropriate 
action is taken. The signaling protocol is also useful for keeping track of session logs and other . 
relevant information for billing and related purposes. It allows sen^ice providers to charge end users 
on a "per consumable unif basis (time, kilobyte etc.) 

The client could potentially include multiple DMIF instances configured to provide specific 
25 services. The user may request the selection of the appropriate DMIF instance. Alternatively, the 
instance may be infen-ed by a DMIF "filter" from the URL requested by the application. The DMIF 
filter, when implemented appropriately, enables the "plug and play" of different DMIF instances 
without the need to reset the tenninal or reconnect the temiinal to the network 

For peer-to-peer applications such as video conferencing, the mobile terminal must also have 
30 some minimal "sen^ei* side functionality so that the coordinating remote server can synchronize and 
manage multiple terminals by issuing appropriate requests. 

9.3 MMS Support: 

MMS[4] provides a means by which multimedia messages (MMs) can be delivered to a User 
35 Agent To support MMS, it is possible for the User Agent (in the case of a VP Client enabled client 
device) Ip mn on either the client device or more appropriately on a server inside the networic. The 
latter capability has several important advantages. It allows the User Agent to support many moiB 
fomriats than would othenwise be feasible. It allows the User Agent to selectively view messages, 
suppr^-stre'ams7lta"Fuafier^ the MMS Relay andthe UA to be closely 
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intertwined and possibly even the same. Several other features required by MMS are also simplified 
and It is Kkely that wireless bandwidth lise would be more judicious and less wasteful using this 
approach. Further the finer grain of control possible with this architecture is likely to make it more 
suitable and functional. 

5 

9 State Reestabllshment and Caching 

The role of state reestabllshment is central to the VP Protocol. State as the protocol defines 
represents the total set of temporarily created infomiafon that can be lost due to a reset or power 
off. The VP Client can then reestablish itself to ite prior state by recontecfing servers and reobtaining 

10 all the infomiafon that they hold cached. The protocol requires sen/ers to hold all the informafon 
necessary for a client to reestablish its prior state following a reset or power off. The procedures 
outlined however have several different steps and actions on the part of the client The VP protocol 
requires servers to cache ALL requests that they sent to a client that the client might need to 
reestablish state on ANY of ite currently active drawables. This each VP Server is responsible for 

15 maintaining state. Further VP Senders also maintain state about event systems as well as sequence 
numbers. 

Further Cliente are mobile. As they move around, their identity (for Instance IP address In 
mobile Internet environmente) might change. However this should not in anyway affect their ability 
to restore state baning one factor, security. The security associated with application access can 

20 change as the VP Client moves. Thus an application that is visible and accessible might be 

available while the VP Client is inside an office (through Wireless LANs or Wide Area Services). 
However it might not be accessible from the outside world. Barring this, the ability to return to 
previously established state is a defining principle. 

However, VP Clients also face the problem of both bandwidth and latency. For this reason. 

25 they tend to cache requests. These caches can persist across disconnections and possibly even 
across power off and power on based on the capabilities of the client device. The aim must be to 
ensure that these caches do not become stale (while the user disconnects). 

A further implementation related Issue relates to the ability of users to browse the cache of 
drawables. This is a recommended feature notably when there is a requirement to browse 

30 infpnmation offline, (for eg. While on the Tokyo subway!). Applications that are inherently data 
Intensive like email and messaging might require support for this. VP Client implementations could 
also support the ability of usere to keep certain caches persistent through user control. This might 
allow a user to retrieve Information that he can then browse as necessary. There is no mandate for 
such a capability. 

35 

10 Supporting Virtual File Stores 

The Virtual File Store existe for allowing users the abliity to access FileSystem and Pereonal 
Space. The feature is completely optional and mainly Intended to bring user content closer to users. 
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Most contemporary client devices, notably those that will be VP Clients will lack much 
pemnanent storage. Few will have disks or other pemfianent storage media. For this reason It is 
important for Users to have a personal file store based inside the network. This Virtual File Store 
could include the users content, applications, etc. The VP Client could access this Inlbnnation either 
5 using a VP Application that allows the user to browse his file Information or by implementing NFS 
V4 [1 3] as part of the VP Client In ttie latter case, the ability could be used to browse information in 
any environment that is local. Further on multiple classmark devices, it could be used to transfer 
applications and associated data to and fipom the network. This will enable effective management of 
storage which is scarce on the client device. 

10 . The Virtual File Store could also be a storehouse of the user's own content As the user 
browses it and requests access (by clicking on files), the file browsing application could launch the 

^ ^ appropriate application that can provide access to the content This could work on remote filesets. 
Further the user could request movement of his filesets to different locations. Further discussion Is 
beyond the scope of this docunient 

15 

Conclusion 

The invention provides a way of supporting larger screen handhelds without burdening their 
Central Processing Unit (CPU) memory and forcing further innovation in storage and battery 
technology. In contrast to WAP, the Invention allows the storage of data primarily in servers in the 

20 Wide Area network and not in the handheld. Further it maintains a log of Ul transactions on the sen/er 
and replays that log. This new use and unique combination of technology is likely to make a complete 
range of applications possible and available that were previously not possible. Further, it will allow 
users to mn several applications concunrently that far exceed the capacities of their cunent handhelds. 
At its core, the Virtual PalmTop uses simple and compound requests and actions on 

25 drawables with the requests being cached by both clients and servers to better use and efficiently 
trade off the constrained handset memory and the limited available bandwidth. Compound requests 
on drawables enables the creation of certain secondary solutions like OnAction primitives, visual 
objects and state reestabiishment through a very small number of remote procedure calls. The 
caching process, the server maintained state process also produces additional benefits in alloviring a. 

30 handset to drop cached drawables as needed and reestablish state on these as the need arises. 

Compound requests was first used in NFS V4 to support wide area file systems where it was 
used to reduce the number of over the wire calls on the Internet In the case of Virtual PalmTop. 
Compound requests serves multiple purposes. It minimizes the number of over the wire calls thus 
reducing latency on drawing actions on drawables. It also reduces the amount of bandwidth used. It 

35 provides an efficient mechanism to update state on the handheld by providing for efficient cache 
updates. It is used In OnAction primitives and visual objects. 

The Mobile Execution Environment (ME}£) has multiple classmarics. The intent of the 
invention is to allow these dassmartcs and other Java and Common Language Infrastmcture (CLI) 
applicatlansisn sWv^ 
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The invention supports the launch of several applications and concunently mn the several 
mechanisms, like the Application List IVIanager (ALM) protocol, reoonnection primitives, and 
recreating state on the client 

The Invention also uses Compound Requests in IMobile networks to ameliorate problems 
5 relating to latency and bandwidth. The compound requests are adapted from ttie pubiidy available 
' Netvrark File System (NFS) V.4 specification (a file access system fo^ 
Workstation) and uses to decrease ttie number of round trips over the wire. Also ttie Event System 
has compound requests to support ttie delivery of line drawing events by ttie user 

Associated witii Compound requests is the notion of "OnAction" primitives which browsers use 
10 ttirbugh scripting languages like European Association for Standardizing Information and 
Communication System (ECMA) Script However, tiiis is a new extension tiiat is not available in X 
Windows that incorporates this into a graphics kemel whose application is based elsewhere. In 
current mobile networks like WAP, ttie browser runs on ttie handset 

However, tiiis invention will allow ttie browser application itself to run on a Company's Server. 
15 As apart of tiie application, the server sends "on action" requests on drawables that tiie client 
caches waiting for user actions. This new innovation brings additional capabilities. 

The invention will decrease ttie bandwidtii usage by maintaining drawable caches tiiat can 
persist such as visual objects and the ability to move them as well as tiie ability to do complex 
actions on drawables. 

20 The invention has a new support for speech and multimodal applications. The document 

describes how devices tiiat use a separate channel or devices ttiat use the data ciiannel in either 
band usage or an out of band protocol can use senders that recognize speech. The recognized 
content can be fed Into multimodal applications running on servers, however mechanisms to enable 
tills on ottier systems are limited. Multimodal applications are complex and ttie World Wide Web 

25 Consortium (W3C) hopes to build WEB standards for ttiem. 

The invention develops feataires that allow tiie user to reconnect to his environment whenever 
he is using ttie ALM and the overall mechanisms for ttie complete server maintained state. This 
achieves a new capability that was previously conceived and defined in ttie Virtual Home . 
Environment VHE (see references) but unrealized ttius far flirough pttier means. 

30 The invention furttier permits offline showing of drawables. When implemented, it allows even 

a well-connected system like ttie Virtual Palmtop to allow users to browse offline. 

Furttier, tiie invention defines a means to allow users ttie ability to not send all substreams In 
a multimedia sti-eam over ttie air while still caching ttiem. 
The invention's main capabilities are ttie following: 

35 (1) Allows server maintained state; 

(2) Allows protocol tiiat allows visual applications running across the mobile networic to 
efficientiy be "viewed" on VP Clients; 

(3) Allows tiie easy reconstruction of user environment and state thus enabling capabilities 
outiinedbyVHE; ' . 
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(4) Allows multiple concurrent applications running on sen/ers to be viewed on handsets; 
and 

Allows a range of devices from very simple to very complicated devices to run complex 
applications like Personal Java (or Classmark II) or CLI (Microsoft C# applications) providing 
5 optimizations for efficient bandwidth use and the minimization of latency on each of them. 
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What is Claimed is: 
. 1 . An apparatus comprising: 

a client computer configured to fit in a person's hand, comprising: 
a central processor unit; 

memory device coupled to the central processor unit, said memory being configured 
.to store instructions to direct the central processing unit; 

input device coupled to the central processor unit 

a communication device coupled to the central processor unit and adapted to. 
establish a wireless communication link with one or more remotely located server computers; and 

a display device coupled to the central processor unit wherein said client computer 
device is adapted to act as a remote output device for one or more application programs running on 
said one or more remotely located server computers without the need for an execution environment 
on the client computer 

2. The apparatus as in claim 1 . wherein the input device is a stylus, a microphone adapted to 
receive speech input, a pointing device. keyt)oard, touch pad. jog dial, joystick, or an infrared input 
device. 

3. The apparatus as in claim 1 . wherein the one or more application programs include one 
active application. 

4. The apparatus as in claim 3, further comprising: 

a portion of the memory device configured as a local cache; wherein drawables 
corresponding to the one or more application programs are stored in the cache for local retrieval and 
display. , 

5. The apparatus as in claim 1 , further comprising: 

first component coupled to the memory device, said first component configured to transmit a 
list of cached drawables for an active application to a server. 

6. The apparatus as in claim 1 , further comprising: 

second component coupled to the memory device, said second component configured to 
receive a compourid request message from the server. 

7. The apparatus as In claim 6. fiirttier comprising; 

third component coupled to the memory device, said third component oonfigurBd to use the 
compound request message to update a display stete of the client computer, 

8. .The apparatus as in claim 1 . further comprising: 

fourth component configured to transmit a user's identification Information to a server; and 
fifth component configured to reoeh^e infonfnation regarding a list of applications previously 
executing for that user. 

9. The apparatus as in claim 1 , further comprising: 

sixth component configured to select one of a plurality of applications from a list of available 
applications. 

-10.--— The-apparatus-asinclalmlrfufther^^ 
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seventh component configured to decode streams of multimedia signals on the client. 

1 1 . The apparatus as in claim 1 0, wherein the seventh component comprises an MPEG 
decoder. 

12. The apparatus as in claim 1, wherein the input device includes: 
5 . a microphone adapted to receive spoken input, and 

a voice-activity detectorrwhereby the voice-activity detector is configured to be activated 
upon the detection of a speech input present at the microphone. 

13. The apparatus as in daim 12, wherein the processor is programmed to instruct the voice- 
activity detector to detect speech input present at the microphone, and transmit the detected 

10 speech. 

14. The apparatus as in daim 1 3, wherein the speech input is directed toward an active 
^. -application. _ 

1 5. The apparatus as in claim 1 3, wherein the speech input is directed toward an application list 
manager (ALM) module, said ALM module is configured to manage an application. 

15 16. An apparatus comprising: 

a client computer configured fit in a person's hand, comprising: 
a central processor unit; 

memory device coupled to the central processor unit, said memory being configured 
to store instructions to direct the central processing unit; 
20 Input device coupled to the central processor unit; 

a communication device coupled to the central processor unit and adapted to 
establish a wireless communication link with one or more remotely located server computers; 

second component coupled to the memory device, said second component configured to 
receive a compound request message from the server; 
25 third component coupled to the memory device, said third component configured to use the 

compound request message to update a display state of the client computer; and 

a display device coupled to the central processor unit, wherein said client computer 
device is adapted to act as a remote output device for one or more application programs running on 
said one or more remotely located server computers over a wide-area mobile network without the 
30 need for an execution environment on the client computer. 

17. The apparatus as in claim 16, wherein the input device is a stylus, a microphone adapted to 
receive speech input, a pointing device, a keyboard.. a touch pad, a jog dial, joystick, or an infrared 
input device. 

35 1 8. The apparatus as in claim 1 7, wherein the one or more application programs include one 
active application. 

19. The apparatus as In claim 16, further comprising: 
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a portion of the memory devioe configured as a local cache; wherein drawables 
corresponding to the one or more application programs are stored in the cache for local retrieval and 
•display. 

20. The apparatus as in daim 16, further comprising: 

5 fourth component configured to transmit a user's identification infomnatlon to a server; and 

fifth component configured to receive information regarding a list of applications previously 
executing for that user. 

21 . The apparatus as in claim 1 6, further comprising: 

sixth component configured to select one of a plurality of applications from a list of available 
10 applications. 

22. The apparatus as in daim 16, ftirther comprising: 

seventh component configured to decode streams of multimedia signals on the client 

23. The apparatus as In daim 22, wherein the seventh component comprises an MPEG 
decoder 

15 24. The apparatus as in claim 16. further comprising: 

first component coupled to the memory device, sard first component configured to transmit a 
list of cached drawables for an active application to a server. 

25. The apparatus as In claim 16, wherein the Input device indudes: 
a microphone adapted to receive spoken input and 

20 a voice-activity detector, whereby the voice-activity detector is configured to be activated 

upon the detection of speech input present at the microphone. 

26. The apparatus as in daim 25, wherein the processor is programmed to instruct the voice- 
activity detedor to detect speech input present at the microphone, and transmit the detected 
speech. 

25 27. The apparatus as in daim 25, wherein the speech input is direded toward an active 
application. 

28. The apparatus as in daim 25, wherein the speech input is directed toward an application list 
manager (ALM) module, said ALM module is configured to manage an application. 
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29. An apparatus comprising: 

a server computer, comprising: 
a central processor unit; 

memory device coupled to tlie central processor unit, said memory being configured 
to store instructions to direct the central processing unit; 

a communication device coupled to the central processor unit and adapted to . - - 
establish a wireless communication linic with one or more remotely located client computers; and 

Instructions stored in the memory device, said instrucKons configured to instruct the 
central processor unit to esteblish a session with a remote client over a wireless communication 
network, execute an application on the server computer, and establish a communication path with 
the remote client such that the remote client is esteblished as an input/output device for the server- 
ain application. _ : . , 

30. A method of esteblishing a client-sen/er communication, said method comprising the steps 
of: 

establishing a session between the client and the server computer, said client and sender 
corriputer being connected using a wireless network; 

executing an application program on the sender computer; 
exporting display of the application program to the client- 
receiving a user input at the server computer; and 
construing the user's input at the server. 

31 . The method as in claim 30, wherein the step of establishing a session between the client 
and the server computer comprises the step of: 

transmitting server system information to the client. 

32. The method as In claim 30, further comprising the step of: 
aggregating a number of requests to be sent to the client; and 
transmitting a compound request to the client 

33. The method as in claim 30, further comprising the step of: 
maintaining a cache of drawables transmitted to the client; and 

replaying the client's stete when the client reesteblishes connection with the server 

34. The method as in dalm 30. wherein the input from the client is receded in a multimodal 
form. 
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35. The method as in daim 30. further compri^ng: 

an event system proxy, said event system proxy receives speech input from Vne dient; 

a speech recognition sen«r. which receives speech input from the event system proxy, 
whereupon the sender computer is configured to interact with the speech recognition server, 
5 construe the speech input at the speech recognition server, and instruct the dient in accordance 
with the construed speech. 

36. The method as in daim 30, further comprising the step of: 
seiectiveiy disabling substreams of audioMsuai date. 

37. The method as in daim 36. further comprising the step of: 

0 receiving an instruction from a user to seiectiveiy disable substreams of audio^/isual date. 



15 - 
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